verification aided regression testing (vart)
DESCRIPTION
ISSTA 2014 Accepted. Verification Aided Regression Testing (VART). Antti E. J. Hyvarinen , Grigory Fedyukovich , Natasha Sharygina University of Lugano. Fabrizio Pastore , Leonardo Mariani University of Milano - Bicocca. Ali Muhammad VTT Finland. Stephan Sehestedt ABB Germany. - PowerPoint PPT PresentationTRANSCRIPT
![Page 1: Verification Aided Regression Testing (VART)](https://reader036.vdocument.in/reader036/viewer/2022062310/56816776550346895ddc705b/html5/thumbnails/1.jpg)
Verification AidedRegression Testing
(VART)Fabrizio Pastore,
Leonardo Mariani University of Milano - Bicocca
Antti E. J. Hyvarinen, Grigory Fedyukovich, Natasha Sharygina
University of Lugano
PINCETTE
Stephan Sehestedt
ABB Germany
Ali MuhammadVTT Finland
ISSTA 2014
Accepted
![Page 2: Verification Aided Regression Testing (VART)](https://reader036.vdocument.in/reader036/viewer/2022062310/56816776550346895ddc705b/html5/thumbnails/2.jpg)
Base Program
Upgraded
Program
working no longer working
Regression Failures
![Page 3: Verification Aided Regression Testing (VART)](https://reader036.vdocument.in/reader036/viewer/2022062310/56816776550346895ddc705b/html5/thumbnails/3.jpg)
Base Program
Upgraded
Program
Test Case
Traditional Regression Testing
Test CaseTest Case
Write tests for Base
![Page 4: Verification Aided Regression Testing (VART)](https://reader036.vdocument.in/reader036/viewer/2022062310/56816776550346895ddc705b/html5/thumbnails/4.jpg)
Base Program
Upgraded
Program
Test Case
Traditional Regression Testing
Test CaseTest Case
Write tests for Base Re-Executeon Upgraded
Execute
![Page 5: Verification Aided Regression Testing (VART)](https://reader036.vdocument.in/reader036/viewer/2022062310/56816776550346895ddc705b/html5/thumbnails/5.jpg)
Base Program
Upgraded
Program
Test Case
Traditional Regression Testing
Test CaseTest Case
Write tests for Base Re-Executeon Upgraded
Execute
PROBLEM: Base Test inputs might not trigger the
regression
![Page 6: Verification Aided Regression Testing (VART)](https://reader036.vdocument.in/reader036/viewer/2022062310/56816776550346895ddc705b/html5/thumbnails/6.jpg)
Monitor + Inference
Base Program
Upgraded
Program
Test Case
Traditional Regression Testing
Test CaseTest Case Execute
Properties Model
Checking
Show PropertiesNot Valid
in Upgraded
![Page 7: Verification Aided Regression Testing (VART)](https://reader036.vdocument.in/reader036/viewer/2022062310/56816776550346895ddc705b/html5/thumbnails/7.jpg)
int isAvailable(node* product){if ( notInitialized(product) )
return 0;if ( product->items > 0 )
return 1;return 0;
}
int isAvailable(node* product){if ( notInitialized(product) )
return 0; if ( product->in_catalog == 0 )
return -1;if ( product->items > 0 )
return 1;return 0;
}long availableProducts( t_store* store ){
node* product = store->products;long total = 0;while ( product != 0 ) {
total += isAvailable( product );
product = product->nxt;
Base Version Upgraded Version
}return total;
Base Version:total >= 0
Upgraded Version:total >= 0 FALSE
![Page 8: Verification Aided Regression Testing (VART)](https://reader036.vdocument.in/reader036/viewer/2022062310/56816776550346895ddc705b/html5/thumbnails/8.jpg)
Dynamic Properties for
Base
Base Program
Upgraded
ProgramTests
for Base
Monitoring + Inference
![Page 9: Verification Aided Regression Testing (VART)](https://reader036.vdocument.in/reader036/viewer/2022062310/56816776550346895ddc705b/html5/thumbnails/9.jpg)
int isAvailable(node* product){if ( notInitialized (product) )
return 0;if ( product->items > 0 )
return 1;return 0;
}
long availableProducts( t_store* store ){node* product = store->products;long total = 0;
while ( product != 0 ) {total += isAvailable( product );product = product->nxt;
}return total;
}
10111213141516
17181920212223242526
![Page 10: Verification Aided Regression Testing (VART)](https://reader036.vdocument.in/reader036/viewer/2022062310/56816776550346895ddc705b/html5/thumbnails/10.jpg)
AcerAspireMacBookAirDell
10111213141516
17181920212223242526
total 0 1 2
return 0 1 1 1
3
return 3
int isAvailable(node* product){if ( notInitialized (product) )
return 0;if ( product->items > 0 )
return 1;return 0;
}
long availableProducts( t_store* store ){node* product = store->products;long total = 0;
while ( product != 0 ) {total += isAvailable( product );product = product->nxt;
}return total;
}
product.items 0 3 2 5product.name MacBookPro
product.items 0 3 2 5
![Page 11: Verification Aided Regression Testing (VART)](https://reader036.vdocument.in/reader036/viewer/2022062310/56816776550346895ddc705b/html5/thumbnails/11.jpg)
AcerAspireMacBookAirDell
10111213141516
17181920212223242526
total 0 1 2
return 0 1 1 1
3
return 3
int isAvailable(node* product){if ( notInitialized (product) )
return 0;if ( product->items > 0 )
return 1;return 0;
}
long availableProducts( t_store* store ){node* product = store->products;long total = 0;
while ( product != 0 ) {total += isAvailable( product );product = product->nxt;
}return total;
}
product.items 0 3 2 5product.name MacBookPro
product.items 0 3 2 5
Inference with Daikon [Ernst et al.]
![Page 12: Verification Aided Regression Testing (VART)](https://reader036.vdocument.in/reader036/viewer/2022062310/56816776550346895ddc705b/html5/thumbnails/12.jpg)
AcerAspireMacBookAirDell
10111213141516
17181920212223242526
total 0 1 2
return 0 1 1 1
3
return 3
int isAvailable(node* product){if ( notInitialized (product) )
return 0;if ( product->items > 0 )
return 1;return 0;
}
long availableProducts( t_store* store ){node* product = store->products;long total = 0;
while ( product != 0 ) {total += isAvailable( product );product = product->nxt;
}return total;
}
product.items >= 0product.name MacBookPro
product.items 0 3 2 5
Inference with Daikon [Ernst et al.]
![Page 13: Verification Aided Regression Testing (VART)](https://reader036.vdocument.in/reader036/viewer/2022062310/56816776550346895ddc705b/html5/thumbnails/13.jpg)
10111213141516
17181920212223242526
total 0 1 2
return one of {0,1}
3
return 3
int isAvailable(node* product){if ( notInitialized (product) )
return 0;if ( product->items > 0 )
return 1;return 0;
}
long availableProducts( t_store* store ){node* product = store->products;long total = 0;
while ( product != 0 ) {total += isAvailable( product );product = product->nxt;
}return total;
}
product.items >= 0
product.items 0 3 2 5
Inference with Daikon [Ernst et al.]
AcerAspireMacBookAirDell
product.name MacBookPro
![Page 14: Verification Aided Regression Testing (VART)](https://reader036.vdocument.in/reader036/viewer/2022062310/56816776550346895ddc705b/html5/thumbnails/14.jpg)
AcerAspireMacBookAirDell
10111213141516
17181920212223242526
total >= 0
return one of {0,1}
return 3
int isAvailable(node* product){if ( notInitialized (product) )
return 0;if ( product->items > 0 )
return 1;return 0;
}
long availableProducts( t_store* store ){node* product = store->products;long total = 0;
while ( product != 0 ) {total += isAvailable( product );product = product->nxt;
}return total;
}
product.items >= 0product.name MacBookPro
product.items 0 3 2 5
Inference with Daikon [Ernst et al.]
![Page 15: Verification Aided Regression Testing (VART)](https://reader036.vdocument.in/reader036/viewer/2022062310/56816776550346895ddc705b/html5/thumbnails/15.jpg)
10111213141516
17181920212223242526
total >= 0
return one of {0,1}
return one of {0,3}
int isAvailable(node* product){if ( notInitialized (product) )
return 0;if ( product->items > 0 )
return 1;return 0;
}
long availableProducts( t_store* store ){node* product = store->products;long total = 0;
while ( product != 0 ) {total += isAvailable( product );product = product->nxt;
}return total;
}
product.items >= 0
product.items >= 0
Inference with Daikon [Ernst et al.]
Overfitting
![Page 16: Verification Aided Regression Testing (VART)](https://reader036.vdocument.in/reader036/viewer/2022062310/56816776550346895ddc705b/html5/thumbnails/16.jpg)
Dynamic Properties for
Base
Monitoring + Inference
Upgraded
ProgramTests
for BaseBase
Program
![Page 17: Verification Aided Regression Testing (VART)](https://reader036.vdocument.in/reader036/viewer/2022062310/56816776550346895ddc705b/html5/thumbnails/17.jpg)
Verified Properties for Base
Dynamic Properties for
Base
Intra-Version Property Verification
Monitoring + Inference
Tests for Base
isAvailable(..)
notInitialized(..)
drawProductBox(..)
main(..)Base
Program
availableItems(..) { . . . while ( .. ) { .. } assert( total > 0 )
Bounded Model Checker
![Page 18: Verification Aided Regression Testing (VART)](https://reader036.vdocument.in/reader036/viewer/2022062310/56816776550346895ddc705b/html5/thumbnails/18.jpg)
Verified Properties for Base
Dynamic Properties for
Base
Intra-Version Property Verification
Monitoring + Inference
Tests for Base
isAvailable(..)
notInitialized(..)
drawProductBox(..)
main(..)Base
Program
availableItems(..) { . . . while ( .. ) { .. } assert( total > 0 )
EvolCheck
function summaries
![Page 19: Verification Aided Regression Testing (VART)](https://reader036.vdocument.in/reader036/viewer/2022062310/56816776550346895ddc705b/html5/thumbnails/19.jpg)
int isAvailable(node* product){if ( notInitialized (product) )
return 0;if ( product->items > 0 )
return 1;return 0;
}
long availableProducts( t_store* store ){node* product = store->products;long total = 0;
while ( product != 0 ) {total += isAvailable( product );product = product->nxt;
}return total;
}
10111213141516
17181920212223242526
total >= 0
return one of {0,1}
return one of {2, 3}
product.items >= 0
![Page 20: Verification Aided Regression Testing (VART)](https://reader036.vdocument.in/reader036/viewer/2022062310/56816776550346895ddc705b/html5/thumbnails/20.jpg)
int isAvailable(node* product){ assert( product->items >= 0 )
if ( notInitialized (product) )return 0;
if ( product->items > 0 )return 1;
return 0;}
long availableProducts( t_store* store ){node* product = store->products;long total = 0;
while ( product != 0 ) {total += isAvailable( product );product = product->nxt;
}return total;
}
17181920212223242526
total >= 0
return one of {0,1}
return one of {2, 3}
10
111213141516
![Page 21: Verification Aided Regression Testing (VART)](https://reader036.vdocument.in/reader036/viewer/2022062310/56816776550346895ddc705b/html5/thumbnails/21.jpg)
long availableProducts( t_store* store ){node* product = store->products;long total = 0;
while ( product != 0 ) {total += isAvailable( product );product = product->nxt;
}return total;
}
17181920212223242526
total >= 0
return one of {2, 3}
int isAvailable(node* product){ assert( product->items >= 0 )
if ( notInitialized (product) ){ int ret_0=0; assert(ret_0 == 0 || ret_0 == 1);
return ret_0; }if ( product->items > 0 )
{ int ret_1=0; assert(ret_1 == 0 || ret_0 == 1); return ret_1; }
int ret_2=0; assert(ret_2 == 0 || ret_2 == 1); return ret_2;}
10
111213141516
![Page 22: Verification Aided Regression Testing (VART)](https://reader036.vdocument.in/reader036/viewer/2022062310/56816776550346895ddc705b/html5/thumbnails/22.jpg)
long availableProducts( t_store* store ){node* product = store->products;long total = 0;
while ( product != 0 ) {total += isAvailable( product );assert( total >= 0 );product = product->nxt;
}return total;
}
171819202122
23242526 return one of {2, 3}
int isAvailable(node* product){ assert( product->items >= 0 )
if ( notInitialized (product) ){ int ret_0=0; assert(ret_0 == 0 || ret_0 == 1);
return ret_0; }if ( product->items > 0 )
{ int ret_1=0; assert(ret_1 == 0 || ret_0 == 1); return ret_1; }
int ret_2=0; assert(ret_2 == 0 || ret_2 == 1); return ret_2;}
10
111213141516
![Page 23: Verification Aided Regression Testing (VART)](https://reader036.vdocument.in/reader036/viewer/2022062310/56816776550346895ddc705b/html5/thumbnails/23.jpg)
long availableProducts( t_store* store ){node* product = store->products;long total = 0;
while ( product != 0 ) {total += isAvailable( product );assert( total >= 0 );product = product->nxt;
}long ret_0 = total; assert (ret_0 == 0 || ret_0 == 3); return
ret_0;}
10
111213141516
171819202122
23242526
int isAvailable(node* product){ assert( product->items >= 0 )
if ( notInitialized (product) ){ int ret_0=0; assert(ret_0 == 0 || ret_0 == 1);
return ret_0; }if ( product->items > 0 )
{ int ret_1=0; assert(ret_1 == 0 || ret_0 == 1); return ret_1; }
int ret_2=0; assert(ret_2 == 0 || ret_2 == 1); return ret_2;}
![Page 24: Verification Aided Regression Testing (VART)](https://reader036.vdocument.in/reader036/viewer/2022062310/56816776550346895ddc705b/html5/thumbnails/24.jpg)
long availableProducts( t_store* store ){node* product = store->products;long total = 0;
while ( product != 0 ) {total += isAvailable( product );assert( total >= 0 );product = product->nxt;
}long ret_0 = total; assert (ret_0 == 0 || ret_0 == 3); return
ret_0;}
171819202122
23242526
Execute Model Checkeron source with assertions
int isAvailable(node* product){ assert( product->items >= 0 )
if ( notInitialized (product) ){ int ret_0=0; assert(ret_0 == 0 || ret_0 == 1);
return ret_0; }if ( product->items > 0 )
{ int ret_1=0; assert(ret_1 == 0 || ret_0 == 1); return ret_1; }
int ret_2=0; assert(ret_2 == 0 || ret_2 == 1); return ret_2;}
10
111213141516
• Verified• Unreachable• False• Unknown
classifies
![Page 25: Verification Aided Regression Testing (VART)](https://reader036.vdocument.in/reader036/viewer/2022062310/56816776550346895ddc705b/html5/thumbnails/25.jpg)
long availableProducts( t_store* store ){node* product = store->products;long total = 0;
while ( product != 0 ) {total += isAvailable( product );assert( total >= 0 );product = product->nxt;
}long ret_0 = total; assert (ret_0 == 0 || ret_0 == 3); return
ret_0;}
171819202122
23242526
int isAvailable(node* product){ assert( product->items >= 0 )
if ( notInitialized (product) ){ int ret_0=0; assert(ret_0 == 0 || ret_0 == 1);
return ret_0; }if ( product->items > 0 )
{ int ret_1=0; assert(ret_1 == 0 || ret_0 == 1); return ret_1; }
int ret_2=0; assert(ret_2 == 0 || ret_2 == 1); return ret_2;}
10
111213141516Execute Model Checkeron source with assertions
• Verified• Unreachable• False• Unknown
classifies
![Page 26: Verification Aided Regression Testing (VART)](https://reader036.vdocument.in/reader036/viewer/2022062310/56816776550346895ddc705b/html5/thumbnails/26.jpg)
Verified Properties for Base
Dynamic Properties for
Base
Intra-Version Property Verification
Monitoring + Inference
Tests for Base
Base Program
![Page 27: Verification Aided Regression Testing (VART)](https://reader036.vdocument.in/reader036/viewer/2022062310/56816776550346895ddc705b/html5/thumbnails/27.jpg)
Verified Properties for Base
Dynamic Properties for
Base
Intra-Version Property Verification
Base Program
Monitoring + Inference
Upgraded
Program
Monitoring + Filtering
Non-Regression Properties
Tests for Base
Tests for Upgrade
Verified Properties for Base
OutdatedProperties
DISCARD
![Page 28: Verification Aided Regression Testing (VART)](https://reader036.vdocument.in/reader036/viewer/2022062310/56816776550346895ddc705b/html5/thumbnails/28.jpg)
long availableProducts( t_store* store ){node* product = store->products;long total = 0;
while ( product != 0 ) {total += isAvailable( product );product = product->nxt;
}return total;
}
10111213141516
18192021222324252627
int isAvailable(node* product){ if ( notInitialized (product) )
return 0; return -1;if ( product->items > 0 )
return 1;return 0;
}
long availableProducts( t_store* store ){node* product = store->products;long total = 0;
while ( product != 0 ) {total += isAvailable( product );product = product->nxt;
}return total;
}
101112131415161718
20212223242526272829
int isAvailable(node* product){ if ( notInitialized (product) )
return 0;if ( notInCatalog (product) )
return -1;if ( product->items > 0 )
return 1;return 0;
}
Base Upgrade
![Page 29: Verification Aided Regression Testing (VART)](https://reader036.vdocument.in/reader036/viewer/2022062310/56816776550346895ddc705b/html5/thumbnails/29.jpg)
long availableProducts( t_store* store ){node* product = store->products;long total = 0;
while ( product != 0 ) {total += isAvailable( product );product = product->nxt;
}return total;
}
101112
13141516
18192021222324252627
int isAvailable(node* product){ if ( notInitialized (product) )
return 0;
return -1;if ( product->items > 0 )
return 1;return 0;
}
long availableProducts( t_store* store ){node* product = store->products;long total = 0;
while ( product != 0 ) {total += isAvailable( product );product = product->nxt;
}return total;
}
101112131415161718
20212223242526272829
int isAvailable(node* product){ if ( notInitialized (product) )
return 0;if ( notInCatalog (product) )
return -1;if ( product->items > 0 )
return 1;return 0;
}
Base Upgrade
![Page 30: Verification Aided Regression Testing (VART)](https://reader036.vdocument.in/reader036/viewer/2022062310/56816776550346895ddc705b/html5/thumbnails/30.jpg)
long availableProducts( t_store* store ){node* product = store->products;long total = 0;
while ( product != 0 ) {total += isAvailable( product );product = product->nxt;
}return total;
}
101112
13141516
int isAvailable(node* product){ if ( notInitialized (product) )
return 0;
return -1;if ( product->items > 0 )
return 1;return 0;
}
long availableProducts( t_store* store ){node* product = store->products;long total = 0;
while ( product != 0 ) {total += isAvailable( product );product = product->nxt;
}return total;
}
101112131415161718
20212223242526272829
int isAvailable(node* product){ if ( notInitialized (product) )
return 0;if ( notInCatalog (product) )
return -1;if ( product->items > 0 )
return 1;return 0;
}
Base Upgrade
return one of {0,1}
product->items >= 0
total >= 0
18192021222324252627
![Page 31: Verification Aided Regression Testing (VART)](https://reader036.vdocument.in/reader036/viewer/2022062310/56816776550346895ddc705b/html5/thumbnails/31.jpg)
long availableProducts( t_store* store ){node* product = store->products;long total = 0;
while ( product != 0 ) {total += isAvailable( product );product = product->nxt;
}return total;
}
101112
13141516
int isAvailable(node* product){ if ( notInitialized (product) )
return 0;
return -1;if ( product->items > 0 )
return 1;return 0;
}
long availableProducts( t_store* store ){node* product = store->products;long total = 0;
while ( product != 0 ) {total += isAvailable( product );product = product->nxt;
}return total;
}
101112131415161718
20212223242526272829
int isAvailable(node* product){ if ( notInitialized (product) )
return 0;if ( notInCatalog (product) )
return -1;if ( product->items > 0 )
return 1;return 0;
}
Base Upgrade
return one of {0,1}
product->items >= 0
total >= 0
18192021222324252627
![Page 32: Verification Aided Regression Testing (VART)](https://reader036.vdocument.in/reader036/viewer/2022062310/56816776550346895ddc705b/html5/thumbnails/32.jpg)
long availableProducts( t_store* store ){node* product = store->products;long total = 0;
while ( product != 0 ) {total += isAvailable( product );product = product->nxt;
}return total;
}
101112131415161718
20212223242526272829
int isAvailable(node* product){ if ( notInitialized (product) )
return 0;if ( notInCatalog (product) )
return -1;if ( product->items > 0 )
return 1;return 0;
}return one of {0,1}
product->items >= 0
total >= 0
void testNotInCatalog(){list_node* prod =
create_product();prod->name = “MacBook”;prod->in_catalog = 0;
assertEquals(-1,isAvailable(prod));}
Test for the upgrade
Upgrade
![Page 33: Verification Aided Regression Testing (VART)](https://reader036.vdocument.in/reader036/viewer/2022062310/56816776550346895ddc705b/html5/thumbnails/33.jpg)
long availableProducts( t_store* store ){node* product = store->products;long total = 0;
while ( product != 0 ) {total += isAvailable( product );product = product->nxt;
}return total;
}
void testNotInCatalog(){list_node* prod =
create_product();prod->name = “MacBook”;prod->in_catalog = 0;
assertEquals(-1,isAvailable(prod));}
Test for the upgrade
101112131415161718
20212223242526272829
int isAvailable(node* product){ if ( notInitialized (product) )
return 0;if ( notInCatalog (product) )
return -1;if ( product->items > 0 )
return 1;return 0;
}return one of {0,1}
product->items >= 0
total >= 0
product.items 0product.in_catalog 0
return -1product.name MacBook
Upgrade
![Page 34: Verification Aided Regression Testing (VART)](https://reader036.vdocument.in/reader036/viewer/2022062310/56816776550346895ddc705b/html5/thumbnails/34.jpg)
Verified Properties for Base
Dynamic Properties for
Base
Intra-Version Property Verification
Base Program
Monitoring + Inference
Monitoring + Filtering
Non-Regression Properties
Tests for Base
Tests for Upgrade
Verified Properties for BaseUpgrade
d Program
![Page 35: Verification Aided Regression Testing (VART)](https://reader036.vdocument.in/reader036/viewer/2022062310/56816776550346895ddc705b/html5/thumbnails/35.jpg)
Verified Properties for Base
Dynamic Properties for
Base
Intra-Version Property Verification
Regression Problems + Counterexamples
Inter-Version Property Verification
Base Program
Monitoring + Inference
Monitoring + Filtering
Non-Regression Properties
Tests for Base
Tests for Upgrade
Verified Properties for BaseUpgrade
d Program
![Page 36: Verification Aided Regression Testing (VART)](https://reader036.vdocument.in/reader036/viewer/2022062310/56816776550346895ddc705b/html5/thumbnails/36.jpg)
long availableProducts( t_store* store ){node* product = store->products;long total = 0;
while ( product != 0 ) {total += isAvailable( product );assert( total >= 0 );product = product->nxt;
}return total;
}
10
1112131415161718
17181920212223242526
int isAvailable(node* product){ assert( product->items >= 0 );
if ( notInitialized (product) )return 0;
if ( notInCatalog (product) )return -1;
if ( product->items > 0 )return 1;
return 0;} classifies
![Page 37: Verification Aided Regression Testing (VART)](https://reader036.vdocument.in/reader036/viewer/2022062310/56816776550346895ddc705b/html5/thumbnails/37.jpg)
long availableProducts( t_store* store ){node* product = store->products;long total = 0;
while ( product != 0 ) {total += isAvailable( product );assert( total >= 0 );product = product->nxt;
}return total;
}
10
1112131415161718
17181920212223242526
int isAvailable(node* product){ assert( product->items >= 0 );
if ( notInitialized (product) )return 0;
if ( notInCatalog (product) )return -1;
if ( product->items > 0 )return 1;
return 0;}
• Verified• False• Unreachable• Unknown
Execute Model Checkeron source with assertionsclassifies
![Page 38: Verification Aided Regression Testing (VART)](https://reader036.vdocument.in/reader036/viewer/2022062310/56816776550346895ddc705b/html5/thumbnails/38.jpg)
long availableProducts( t_store* store ){node* product = store->products;long total = 0;
while ( product != 0 ) {total += isAvailable( product );assert( total >= 0 );product = product->nxt;
}return total;
}
10
1112131415161718
17181920212223242526
int isAvailable(node* product){ assert( product->items >= 0 );
if ( notInitialized (product) )return 0;
if ( notInCatalog (product) )return -1;
if ( product->items > 0 )return 1;
return 0;}
• Verified• False• Unreachable• Unknown
Execute Model Checkeron source with assertionsclassifies
FALSE
Presented to developers
![Page 39: Verification Aided Regression Testing (VART)](https://reader036.vdocument.in/reader036/viewer/2022062310/56816776550346895ddc705b/html5/thumbnails/39.jpg)
long availableProducts( t_store* store ){node* product = store->products;long total = 0;
while ( product != 0 ) {total += isAvailable( product );assert( total >= 0 );product = product->nxt;
}return total;
}
10
1112131415161718
17181920212223242526
int isAvailable(node* product){ assert( product->items >= 0 );
if ( notInitialized (product) )return 0;
if ( notInCatalog (product) )return -1;
if ( product->items > 0 )return 1;
return 0;}
• Verified• False• Unreachable• Unknown
Execute Model Checkeron source with assertionsclassifies
FALSE
Presented to developers
![Page 40: Verification Aided Regression Testing (VART)](https://reader036.vdocument.in/reader036/viewer/2022062310/56816776550346895ddc705b/html5/thumbnails/40.jpg)
Dynamic Properties for
Base
Verified Properties for Base
Intra-Version Property Verification
Base Program
Monitoring + Inference
Upgraded
ProgramTests
for Base
Regression Problems + Counterexamples
Inter-Version Property Verification
Monitoring + Filtering
Non-Regression Properties
Tests for Upgrade
Verified Properties for Base
![Page 41: Verification Aided Regression Testing (VART)](https://reader036.vdocument.in/reader036/viewer/2022062310/56816776550346895ddc705b/html5/thumbnails/41.jpg)
Console Version Eclipse GUI
Tool ImplementationWWW.LTA.DISCO.UNIMIB.IT/TOOLS/VART
![Page 42: Verification Aided Regression Testing (VART)](https://reader036.vdocument.in/reader036/viewer/2022062310/56816776550346895ddc705b/html5/thumbnails/42.jpg)
Empirical Validation
• Effectiveness wrt Test Coverage• Effectiveness with Regression
Faults not covered by Test Suites
![Page 43: Verification Aided Regression Testing (VART)](https://reader036.vdocument.in/reader036/viewer/2022062310/56816776550346895ddc705b/html5/thumbnails/43.jpg)
Effectiveness wrt Test Coverage
High CoverageLow CoverageComplete PropertiesExtensive Tests
Poor PropertiesPoor Tests
![Page 44: Verification Aided Regression Testing (VART)](https://reader036.vdocument.in/reader036/viewer/2022062310/56816776550346895ddc705b/html5/thumbnails/44.jpg)
Effectiveness vs Test Coverage
• GREP, 871 Test Cases• Faulty Versions of GREP from
Software-artifacts Infrastructure Repository (SIR) http://sir.unl.edu
• Selected all the faulty versions derived from GREP v3 and v4, with faults covered by test cases
High CoverageLow CoverageComplete PropertiesExtensive Tests
Poor PropertiesPoor Tests
![Page 45: Verification Aided Regression Testing (VART)](https://reader036.vdocument.in/reader036/viewer/2022062310/56816776550346895ddc705b/html5/thumbnails/45.jpg)
Fault ID
Base Version Upgrade Version Change Size
v3 DG10
10/6/1999 16:44:13
10/6/1999 1:13:21
483
v3 KP2v3 KP9v3 DG1 8/13/1999
14:45:218/13/1999 15:02:00
944v3 DG8v3 KP7v3 DG3 10/07/1999
3:42:5710/12/1999
4:11:40185
v3 DG2v4 KP8 1/21/2000
2:22:471/25/2000
3:34:23354
v4 DG3v4 KP6 1/17/2000
0:55:061/20/2000
4:43:0389
![Page 46: Verification Aided Regression Testing (VART)](https://reader036.vdocument.in/reader036/viewer/2022062310/56816776550346895ddc705b/html5/thumbnails/46.jpg)
Fault ID
Base Version Upgrade Version Change Size
v3 DG10
10/6/1999 16:44:13
10/6/1999 1:13:21
483
v3 KP2v3 KP9v3 DG1 8/13/1999
14:45:218/13/1999 15:02:00
944v3 DG8v3 KP7v3 DG3 10/07/1999
3:42:5710/12/1999
4:11:40185
v3 DG2v4 KP8 1/21/2000
2:22:471/25/2000
3:34:23354
v4 DG3v4 KP6 1/17/2000
0:55:061/20/2000
4:43:0389
![Page 47: Verification Aided Regression Testing (VART)](https://reader036.vdocument.in/reader036/viewer/2022062310/56816776550346895ddc705b/html5/thumbnails/47.jpg)
TC1
TC2 TC3 TC4 ... TC817
![Page 48: Verification Aided Regression Testing (VART)](https://reader036.vdocument.in/reader036/viewer/2022062310/56816776550346895ddc705b/html5/thumbnails/48.jpg)
Functions
TC1
TC2 TC3 TC4
... TC817
copy(..) branch1
✔
branch2
✔
branch3
✔ ✔ ✔
lex(..) branch1
✔
branch2
✔ ✔ ✔
branch3
✔
main(..) branch1
✔ ✔ ✔
branch2
...
MINIMAL REGRESSION TEST SUITE
Mod
ified
![Page 49: Verification Aided Regression Testing (VART)](https://reader036.vdocument.in/reader036/viewer/2022062310/56816776550346895ddc705b/html5/thumbnails/49.jpg)
Incr
emen
tal C
over
age
20%
50%
20% 50% MRT(75%)
TC1 TC817 TC3 TC.. TC..TC..
![Page 50: Verification Aided Regression Testing (VART)](https://reader036.vdocument.in/reader036/viewer/2022062310/56816776550346895ddc705b/html5/thumbnails/50.jpg)
Coverage Identified Faults
Property Violations
Fault ID RegressionTesting
VART
TP FP
Coverage 20%
3 5 5 0
Coverage 50%
7 8 2 0
Minimal Regression Test Suite
10 10 0 0(coverage 75%)
![Page 51: Verification Aided Regression Testing (VART)](https://reader036.vdocument.in/reader036/viewer/2022062310/56816776550346895ddc705b/html5/thumbnails/51.jpg)
Effectiveness with Regression Faults Not Covered by Test Suites
![Page 52: Verification Aided Regression Testing (VART)](https://reader036.vdocument.in/reader036/viewer/2022062310/56816776550346895ddc705b/html5/thumbnails/52.jpg)
Subject
Size (LOC)
Base Upgrade Test Suite Size
VTT 488 10.1 10.2 1000
VTT: Motion Trajectory Controller
Effectiveness with Regression Faults Not Covered by Test Suites
![Page 53: Verification Aided Regression Testing (VART)](https://reader036.vdocument.in/reader036/viewer/2022062310/56816776550346895ddc705b/html5/thumbnails/53.jpg)
Subject
Size (LOC)
Base Upgrade Test Suite Size
VTT 488 10.1 10.2 1000ABB-1 200 - - 1600ABB-2 200 - - 1600ABB-3 200 - - 1600
ABB: Protection Function
Effectiveness with Regression Faults Not Covered by Test Suites
![Page 54: Verification Aided Regression Testing (VART)](https://reader036.vdocument.in/reader036/viewer/2022062310/56816776550346895ddc705b/html5/thumbnails/54.jpg)
Subject
Size (LOC)
Base Upgrade Test Suite Size
VTT 488 10.1 10.2 1000ABB-1 200 - - 1600ABB-2 200 - - 1600ABB-3 200 - - 1600SORT 4653 20/06/2012 02/07/2012 427GREP 590 01/17/2001 01/20/2001 817
Effectiveness with Regression Faults Not Covered by Test Suites
![Page 55: Verification Aided Regression Testing (VART)](https://reader036.vdocument.in/reader036/viewer/2022062310/56816776550346895ddc705b/html5/thumbnails/55.jpg)
Subject DynamicProperties
Non-RegressionProperties
Property ViolationsTP FP Precisio
n
VTT 1045 658 15 0 1.00ABB-1 3278 163 57 1 0.98ABB-2 3278 163 57 1 0.98ABB-3 3278 163 0 1 0.00SORT 356 2 1 0 1.00GREP 3303 51 3 0 1.00
![Page 56: Verification Aided Regression Testing (VART)](https://reader036.vdocument.in/reader036/viewer/2022062310/56816776550346895ddc705b/html5/thumbnails/56.jpg)
Pre- and Post-conditionscatch illegal uses of
functions
![Page 57: Verification Aided Regression Testing (VART)](https://reader036.vdocument.in/reader036/viewer/2022062310/56816776550346895ddc705b/html5/thumbnails/57.jpg)
New code with wrong use of base function
stream_open (char const *file, . . . if (*how == 'r’) { . . . } return fopen (file, how);}
stream_open (char const *file, . . . if (*how == 'r’) { . . . } else if (*how == 'w’) { if (ftruncate (outfd, 0) != 0)
error (EXIT_FAILURE, . . . fp = fdopen (outfd, how); } else return fp;}
SORT_FAILURE
void error (int status, . . . assert( status == 0 || status == 2 );
Base Upgrade
![Page 58: Verification Aided Regression Testing (VART)](https://reader036.vdocument.in/reader036/viewer/2022062310/56816776550346895ddc705b/html5/thumbnails/58.jpg)
Unreachable assertionscatch errors in branch
conditions
![Page 59: Verification Aided Regression Testing (VART)](https://reader036.vdocument.in/reader036/viewer/2022062310/56816776550346895ddc705b/html5/thumbnails/59.jpg)
Missing else: unreachable assertion
if (ISDIGIT(c)) { minrep = c - '0'; for (;;) { ... minrep = 10 * minrep + c - '0'; }}
else if (c == ',’) dfaerror(_("malformed")); if (c == ',’){for (;;) {
if (ISDIGIT(c)) { minrep = c - '0'; for (;;) { ... minrep = 10 * minrep + c - '0'; }}
else dfaerror(_("malformed")); if (c == ',’) { . . . for (;;) {
assert( c >= 44 );
Base Upgrade
![Page 60: Verification Aided Regression Testing (VART)](https://reader036.vdocument.in/reader036/viewer/2022062310/56816776550346895ddc705b/html5/thumbnails/60.jpg)
Assertions identifythe variables with wrong
assignments
![Page 61: Verification Aided Regression Testing (VART)](https://reader036.vdocument.in/reader036/viewer/2022062310/56816776550346895ddc705b/html5/thumbnails/61.jpg)
Wrong Refactoring
status = count;if (list_files == 1 - 2 * status) printf ("%s%c", filename, '\n' & filename_mask);
if (count) { status = 0; if (list_files == 1) printf ("%s\n", filename);} else { status = 1; if (list_files == -1) printf ("%s\n", filename);} assert( count != status );
assert( status <= 1 );assert( desc >= 0 );
! missing
Base Upgrade
![Page 62: Verification Aided Regression Testing (VART)](https://reader036.vdocument.in/reader036/viewer/2022062310/56816776550346895ddc705b/html5/thumbnails/62.jpg)
![Page 63: Verification Aided Regression Testing (VART)](https://reader036.vdocument.in/reader036/viewer/2022062310/56816776550346895ddc705b/html5/thumbnails/63.jpg)
Verification Aided Regression Testing (VART)
Compensate Low Test Coverage
Spot Regressions Hard to Find
http://www.lta.disco.unimib.it/tools/vart/