![Page 1: SmarTest - isa.us.es · Testing real highly-configurable tools Artificial models to test Ana B. Sánchez, Sergio Segura, Jose A. Parejo and Antonio Ruiz Cortés. Variability Testing](https://reader033.vdocument.in/reader033/viewer/2022050105/5f43727e27ed8b3bb47cabee/html5/thumbnails/1.jpg)
![Page 2: SmarTest - isa.us.es · Testing real highly-configurable tools Artificial models to test Ana B. Sánchez, Sergio Segura, Jose A. Parejo and Antonio Ruiz Cortés. Variability Testing](https://reader033.vdocument.in/reader033/viewer/2022050105/5f43727e27ed8b3bb47cabee/html5/thumbnails/2.jpg)
SmarTest: Accelerating the detection of faults in Drupal
Ana Belén Sánchez
http://www.isa.us.es/members/anabelen.sanchez
![Page 3: SmarTest - isa.us.es · Testing real highly-configurable tools Artificial models to test Ana B. Sánchez, Sergio Segura, Jose A. Parejo and Antonio Ruiz Cortés. Variability Testing](https://reader033.vdocument.in/reader033/viewer/2022050105/5f43727e27ed8b3bb47cabee/html5/thumbnails/3.jpg)
Ana Belén Sánchez, Sergio Segura, Antonio Ruiz-Cortés, Gabriel Hidalgo
Research Group of Applied Software Engineering
(www.isa.us.es)
Escuela Técnica Superior de Ingeniería Informática University of Seville
I am a researcher and a teacher
![Page 4: SmarTest - isa.us.es · Testing real highly-configurable tools Artificial models to test Ana B. Sánchez, Sergio Segura, Jose A. Parejo and Antonio Ruiz Cortés. Variability Testing](https://reader033.vdocument.in/reader033/viewer/2022050105/5f43727e27ed8b3bb47cabee/html5/thumbnails/4.jpg)
![Page 5: SmarTest - isa.us.es · Testing real highly-configurable tools Artificial models to test Ana B. Sánchez, Sergio Segura, Jose A. Parejo and Antonio Ruiz Cortés. Variability Testing](https://reader033.vdocument.in/reader033/viewer/2022050105/5f43727e27ed8b3bb47cabee/html5/thumbnails/5.jpg)
![Page 6: SmarTest - isa.us.es · Testing real highly-configurable tools Artificial models to test Ana B. Sánchez, Sergio Segura, Jose A. Parejo and Antonio Ruiz Cortés. Variability Testing](https://reader033.vdocument.in/reader033/viewer/2022050105/5f43727e27ed8b3bb47cabee/html5/thumbnails/6.jpg)
Why am I here, in a DrupalCon?
Ph.D. in automated testing on variability-intensive systems
![Page 7: SmarTest - isa.us.es · Testing real highly-configurable tools Artificial models to test Ana B. Sánchez, Sergio Segura, Jose A. Parejo and Antonio Ruiz Cortés. Variability Testing](https://reader033.vdocument.in/reader033/viewer/2022050105/5f43727e27ed8b3bb47cabee/html5/thumbnails/7.jpg)
Variability-intensive systems
![Page 8: SmarTest - isa.us.es · Testing real highly-configurable tools Artificial models to test Ana B. Sánchez, Sergio Segura, Jose A. Parejo and Antonio Ruiz Cortés. Variability Testing](https://reader033.vdocument.in/reader033/viewer/2022050105/5f43727e27ed8b3bb47cabee/html5/thumbnails/8.jpg)
Variability-intensive systems
Mandatory Optional
Alternative (choose 1) Or
Excludes
Requires
A Feature Model represents all the possible configurations of the Variability-intensive system in terms of features and relationships among them.
= {MobilePhone, Calls, Screen, High resolution, Media, Camera}
Feature
![Page 9: SmarTest - isa.us.es · Testing real highly-configurable tools Artificial models to test Ana B. Sánchez, Sergio Segura, Jose A. Parejo and Antonio Ruiz Cortés. Variability Testing](https://reader033.vdocument.in/reader033/viewer/2022050105/5f43727e27ed8b3bb47cabee/html5/thumbnails/9.jpg)
![Page 10: SmarTest - isa.us.es · Testing real highly-configurable tools Artificial models to test Ana B. Sánchez, Sergio Segura, Jose A. Parejo and Antonio Ruiz Cortés. Variability Testing](https://reader033.vdocument.in/reader033/viewer/2022050105/5f43727e27ed8b3bb47cabee/html5/thumbnails/10.jpg)
Testing problems
Too many configurations!
Time-consuming
![Page 11: SmarTest - isa.us.es · Testing real highly-configurable tools Artificial models to test Ana B. Sánchez, Sergio Segura, Jose A. Parejo and Antonio Ruiz Cortés. Variability Testing](https://reader033.vdocument.in/reader033/viewer/2022050105/5f43727e27ed8b3bb47cabee/html5/thumbnails/11.jpg)
Testing variability-intensive systems
Test case selection techniques reduce the test space by selecting a portion of the configurations to be tested.
![Page 12: SmarTest - isa.us.es · Testing real highly-configurable tools Artificial models to test Ana B. Sánchez, Sergio Segura, Jose A. Parejo and Antonio Ruiz Cortés. Variability Testing](https://reader033.vdocument.in/reader033/viewer/2022050105/5f43727e27ed8b3bb47cabee/html5/thumbnails/12.jpg)
Testing variability-intensive systems
Test case prioritization techniques schedule test cases for execution in an order that attempts to increase their effectiveness at meeting some performance goal.
![Page 13: SmarTest - isa.us.es · Testing real highly-configurable tools Artificial models to test Ana B. Sánchez, Sergio Segura, Jose A. Parejo and Antonio Ruiz Cortés. Variability Testing](https://reader033.vdocument.in/reader033/viewer/2022050105/5f43727e27ed8b3bb47cabee/html5/thumbnails/13.jpg)
Testing variability-intensive systems
How to evaluate these testing techniques?
Artificial models to test
![Page 14: SmarTest - isa.us.es · Testing real highly-configurable tools Artificial models to test Ana B. Sánchez, Sergio Segura, Jose A. Parejo and Antonio Ruiz Cortés. Variability Testing](https://reader033.vdocument.in/reader033/viewer/2022050105/5f43727e27ed8b3bb47cabee/html5/thumbnails/14.jpg)
![Page 15: SmarTest - isa.us.es · Testing real highly-configurable tools Artificial models to test Ana B. Sánchez, Sergio Segura, Jose A. Parejo and Antonio Ruiz Cortés. Variability Testing](https://reader033.vdocument.in/reader033/viewer/2022050105/5f43727e27ed8b3bb47cabee/html5/thumbnails/15.jpg)
Towards real highly-configurable tools
Artificial models to test
Studied Tools
![Page 16: SmarTest - isa.us.es · Testing real highly-configurable tools Artificial models to test Ana B. Sánchez, Sergio Segura, Jose A. Parejo and Antonio Ruiz Cortés. Variability Testing](https://reader033.vdocument.in/reader033/viewer/2022050105/5f43727e27ed8b3bb47cabee/html5/thumbnails/16.jpg)
Testing real highly-configurable tools
1. Available code
2. Detailed fault reports
3. Automated test cases
4. Extensive documentation
5. > 630,000 users and developers
6. > 30,000 modules
![Page 17: SmarTest - isa.us.es · Testing real highly-configurable tools Artificial models to test Ana B. Sánchez, Sergio Segura, Jose A. Parejo and Antonio Ruiz Cortés. Variability Testing](https://reader033.vdocument.in/reader033/viewer/2022050105/5f43727e27ed8b3bb47cabee/html5/thumbnails/17.jpg)
Testing real highly-configurable tools
Artificial models to test
Ana B. Sánchez, Sergio Segura, Jose A. Parejo and Antonio Ruiz Cortés. Variability Testing in the Wild: The Drupal Case Study Published in Software and Systems Modeling Journal, 2015
(http://www.isa.us.es/publications/type/article-journal/9998/variability-testing-wild-drupal-case-study)
![Page 18: SmarTest - isa.us.es · Testing real highly-configurable tools Artificial models to test Ana B. Sánchez, Sergio Segura, Jose A. Parejo and Antonio Ruiz Cortés. Variability Testing](https://reader033.vdocument.in/reader033/viewer/2022050105/5f43727e27ed8b3bb47cabee/html5/thumbnails/18.jpg)
Our approach: The Drupal case study
Artificial models to test
Module .info files
Drupal documentation
Module JIT
![Page 19: SmarTest - isa.us.es · Testing real highly-configurable tools Artificial models to test Ana B. Sánchez, Sergio Segura, Jose A. Parejo and Antonio Ruiz Cortés. Variability Testing](https://reader033.vdocument.in/reader033/viewer/2022050105/5f43727e27ed8b3bb47cabee/html5/thumbnails/19.jpg)
The Drupal variability model Drupal 7.23
Node Filter CtoolsFile
Options
Path
Blog Forum Ctools access ruleset
Ctools custom content
Views content
Field Field UI
Field SQL storage
ImageSystem UserViews
Views UITaxonomy
Comment
Text
Date
Token
Forum Requires Taxonomy Views content Requires Views Panels IPE Requires Ctools Rules Requires Entity tokens Forum Requires Options Taxonomy Requires Options Panel Nodes Requires Ctools Rules scheduler Requires Entity APIForum Requires Comment Date views Requires Ctools Pathauto Requires Token Rules scheduler Requires Entity tokensImage Requires File Date views Requires Views Pathauto Requires Path Rules UI Requires Entity API Views Requires Ctools Panels Requires Ctools Rules Requires Entity API Rules UI Requires Entity tokens Views UI Requires Ctools
PathautoIMCE
Date views
Date popup
Date API
WebForm
Link EntityAPI
Entitytokens
CKEditor Captcha
ImageCaptcha
Features Panels
Panel nodes
Panels IPE
LibrariesAPI
JqueryUpdate
GoogleAnalytics
Rules
Rules UIRulesscheduler
Backupmigrate
MandatoryOptional
48 modules 21 dependencies 2,000 millions of configurations
![Page 20: SmarTest - isa.us.es · Testing real highly-configurable tools Artificial models to test Ana B. Sánchez, Sergio Segura, Jose A. Parejo and Antonio Ruiz Cortés. Variability Testing](https://reader033.vdocument.in/reader033/viewer/2022050105/5f43727e27ed8b3bb47cabee/html5/thumbnails/20.jpg)
Drupal information
LoC
Module Size
Changes Commits
Tests 352 test cases 24152 assertions
![Page 21: SmarTest - isa.us.es · Testing real highly-configurable tools Artificial models to test Ana B. Sánchez, Sergio Segura, Jose A. Parejo and Antonio Ruiz Cortés. Variability Testing](https://reader033.vdocument.in/reader033/viewer/2022050105/5f43727e27ed8b3bb47cabee/html5/thumbnails/21.jpg)
Drupal information
CC
Cyclomatic Complexity
Drupal Installations
Drupal Contributors
![Page 22: SmarTest - isa.us.es · Testing real highly-configurable tools Artificial models to test Ana B. Sánchez, Sergio Segura, Jose A. Parejo and Antonio Ruiz Cortés. Variability Testing](https://reader033.vdocument.in/reader033/viewer/2022050105/5f43727e27ed8b3bb47cabee/html5/thumbnails/22.jpg)
Drupal faults
Bug tracking System
3.232 Faults in v7.23
160 integration faults:
• 132 caused by 2 modules
• 25 caused by 3 modules
• 3 caused by 4 modules
![Page 23: SmarTest - isa.us.es · Testing real highly-configurable tools Artificial models to test Ana B. Sánchez, Sergio Segura, Jose A. Parejo and Antonio Ruiz Cortés. Variability Testing](https://reader033.vdocument.in/reader033/viewer/2022050105/5f43727e27ed8b3bb47cabee/html5/thumbnails/23.jpg)
Statistical study
Module Size Faults Module
Changes Faults
Faults v7.22
Faults v7.23
Module CC Faults
C1 C2
C3 C4
![Page 24: SmarTest - isa.us.es · Testing real highly-configurable tools Artificial models to test Ana B. Sánchez, Sergio Segura, Jose A. Parejo and Antonio Ruiz Cortés. Variability Testing](https://reader033.vdocument.in/reader033/viewer/2022050105/5f43727e27ed8b3bb47cabee/html5/thumbnails/24.jpg)
Statistical study
Module Developers
Faults Tests ? Faults?
Optional Modules Faults
91 %
Core Mandatory Modules Faults
41 %
C5 C6
![Page 25: SmarTest - isa.us.es · Testing real highly-configurable tools Artificial models to test Ana B. Sánchez, Sergio Segura, Jose A. Parejo and Antonio Ruiz Cortés. Variability Testing](https://reader033.vdocument.in/reader033/viewer/2022050105/5f43727e27ed8b3bb47cabee/html5/thumbnails/25.jpg)
Test case prioritization criteria
Size–driven Criterion
Fault–driven Criterion
CC-driven Criterion
Change–driven Criterion
Priori%za%on criteria based on previous sta%s%cal study
![Page 26: SmarTest - isa.us.es · Testing real highly-configurable tools Artificial models to test Ana B. Sánchez, Sergio Segura, Jose A. Parejo and Antonio Ruiz Cortés. Variability Testing](https://reader033.vdocument.in/reader033/viewer/2022050105/5f43727e27ed8b3bb47cabee/html5/thumbnails/26.jpg)
![Page 27: SmarTest - isa.us.es · Testing real highly-configurable tools Artificial models to test Ana B. Sánchez, Sergio Segura, Jose A. Parejo and Antonio Ruiz Cortés. Variability Testing](https://reader033.vdocument.in/reader033/viewer/2022050105/5f43727e27ed8b3bb47cabee/html5/thumbnails/27.jpg)
Test case prioritization criteria
94.2 % Test case selection
Fault-driven Prioritization
2.
Could we accelerate the detection of faults in Drupal using previous information?
86.2 % Test case selection
1.
![Page 28: SmarTest - isa.us.es · Testing real highly-configurable tools Artificial models to test Ana B. Sánchez, Sergio Segura, Jose A. Parejo and Antonio Ruiz Cortés. Variability Testing](https://reader033.vdocument.in/reader033/viewer/2022050105/5f43727e27ed8b3bb47cabee/html5/thumbnails/28.jpg)
![Page 29: SmarTest - isa.us.es · Testing real highly-configurable tools Artificial models to test Ana B. Sánchez, Sergio Segura, Jose A. Parejo and Antonio Ruiz Cortés. Variability Testing](https://reader033.vdocument.in/reader033/viewer/2022050105/5f43727e27ed8b3bb47cabee/html5/thumbnails/29.jpg)
Applying our proposal to Drupal
SimpleTest
![Page 30: SmarTest - isa.us.es · Testing real highly-configurable tools Artificial models to test Ana B. Sánchez, Sergio Segura, Jose A. Parejo and Antonio Ruiz Cortés. Variability Testing](https://reader033.vdocument.in/reader033/viewer/2022050105/5f43727e27ed8b3bb47cabee/html5/thumbnails/30.jpg)
Anayzing the SimpleTest module
Main view SimpleTest
![Page 31: SmarTest - isa.us.es · Testing real highly-configurable tools Artificial models to test Ana B. Sánchez, Sergio Segura, Jose A. Parejo and Antonio Ruiz Cortés. Variability Testing](https://reader033.vdocument.in/reader033/viewer/2022050105/5f43727e27ed8b3bb47cabee/html5/thumbnails/31.jpg)
Our proposal: SmarTest
Main view SmarTest
![Page 32: SmarTest - isa.us.es · Testing real highly-configurable tools Artificial models to test Ana B. Sánchez, Sergio Segura, Jose A. Parejo and Antonio Ruiz Cortés. Variability Testing](https://reader033.vdocument.in/reader033/viewer/2022050105/5f43727e27ed8b3bb47cabee/html5/thumbnails/32.jpg)
Our proposal: SmarTest
![Page 33: SmarTest - isa.us.es · Testing real highly-configurable tools Artificial models to test Ana B. Sánchez, Sergio Segura, Jose A. Parejo and Antonio Ruiz Cortés. Variability Testing](https://reader033.vdocument.in/reader033/viewer/2022050105/5f43727e27ed8b3bb47cabee/html5/thumbnails/33.jpg)
Our proposal: SmarTest
![Page 34: SmarTest - isa.us.es · Testing real highly-configurable tools Artificial models to test Ana B. Sánchez, Sergio Segura, Jose A. Parejo and Antonio Ruiz Cortés. Variability Testing](https://reader033.vdocument.in/reader033/viewer/2022050105/5f43727e27ed8b3bb47cabee/html5/thumbnails/34.jpg)
Our proposal: SmarTest
Test Prioritization SmarTest
![Page 35: SmarTest - isa.us.es · Testing real highly-configurable tools Artificial models to test Ana B. Sánchez, Sergio Segura, Jose A. Parejo and Antonio Ruiz Cortés. Variability Testing](https://reader033.vdocument.in/reader033/viewer/2022050105/5f43727e27ed8b3bb47cabee/html5/thumbnails/35.jpg)
Our proposal: SmarTest
Continuos feedback in real time SmartTest
![Page 36: SmarTest - isa.us.es · Testing real highly-configurable tools Artificial models to test Ana B. Sánchez, Sergio Segura, Jose A. Parejo and Antonio Ruiz Cortés. Variability Testing](https://reader033.vdocument.in/reader033/viewer/2022050105/5f43727e27ed8b3bb47cabee/html5/thumbnails/36.jpg)
Our proposal: SmarTest
![Page 37: SmarTest - isa.us.es · Testing real highly-configurable tools Artificial models to test Ana B. Sánchez, Sergio Segura, Jose A. Parejo and Antonio Ruiz Cortés. Variability Testing](https://reader033.vdocument.in/reader033/viewer/2022050105/5f43727e27ed8b3bb47cabee/html5/thumbnails/37.jpg)
Our proposal: SmarTest
![Page 38: SmarTest - isa.us.es · Testing real highly-configurable tools Artificial models to test Ana B. Sánchez, Sergio Segura, Jose A. Parejo and Antonio Ruiz Cortés. Variability Testing](https://reader033.vdocument.in/reader033/viewer/2022050105/5f43727e27ed8b3bb47cabee/html5/thumbnails/38.jpg)
![Page 39: SmarTest - isa.us.es · Testing real highly-configurable tools Artificial models to test Ana B. Sánchez, Sergio Segura, Jose A. Parejo and Antonio Ruiz Cortés. Variability Testing](https://reader033.vdocument.in/reader033/viewer/2022050105/5f43727e27ed8b3bb47cabee/html5/thumbnails/39.jpg)
Conclusions
¿What improvements do we add to SimpleTest?
SimpleTest ü Designed to make testing easier
ü A powerful automated testing
ü Selector to select tests to execute
ü Detailed inform. about test execution
ü Possibility to follow the trace of errors
SmarTest ü Customizable Dashboard with run-time extracted data to guide the testing
ü Test prioritization to detect faults faster
ü Automated testing with feedback in real time
ü Time-out option to automatically stop the test execution
![Page 40: SmarTest - isa.us.es · Testing real highly-configurable tools Artificial models to test Ana B. Sánchez, Sergio Segura, Jose A. Parejo and Antonio Ruiz Cortés. Variability Testing](https://reader033.vdocument.in/reader033/viewer/2022050105/5f43727e27ed8b3bb47cabee/html5/thumbnails/40.jpg)
Conclusions
Things that we are working on …
• Add new widgets to the dashboard.
• Include multi-prioritization criteria (based on genetic algorithms to detect faults faster).
• Study the integration of SmarTest with other test systems: Behat, phpUnit… • Get SmarTest is accepted in Drupal. Now is in Sandbox: https://www.drupal.org/sandbox/annasan/2503695
• ¿Any suggestions?
![Page 41: SmarTest - isa.us.es · Testing real highly-configurable tools Artificial models to test Ana B. Sánchez, Sergio Segura, Jose A. Parejo and Antonio Ruiz Cortés. Variability Testing](https://reader033.vdocument.in/reader033/viewer/2022050105/5f43727e27ed8b3bb47cabee/html5/thumbnails/41.jpg)
Acknowledgments
This work was partially supported by the European Commission (FEDER), the Spanish and the Andalusian R&D&I programmes (Grants IPT-2012-0890-390000 (SAAS FIRE- 46 WALL), TIN2012-32273 (TAPAS), TIC-5906 (THEOS), TIC-1867 (COPAS)).
![Page 42: SmarTest - isa.us.es · Testing real highly-configurable tools Artificial models to test Ana B. Sánchez, Sergio Segura, Jose A. Parejo and Antonio Ruiz Cortés. Variability Testing](https://reader033.vdocument.in/reader033/viewer/2022050105/5f43727e27ed8b3bb47cabee/html5/thumbnails/42.jpg)
![Page 43: SmarTest - isa.us.es · Testing real highly-configurable tools Artificial models to test Ana B. Sánchez, Sergio Segura, Jose A. Parejo and Antonio Ruiz Cortés. Variability Testing](https://reader033.vdocument.in/reader033/viewer/2022050105/5f43727e27ed8b3bb47cabee/html5/thumbnails/43.jpg)
“Research is to see what everybody else has seen, and to think what nobody else has thought".
– Albert Szent-Gyorgyi, 1893–1986 Hungarian Biochemist, 1937 Nobel Prize for Medicine
![Page 44: SmarTest - isa.us.es · Testing real highly-configurable tools Artificial models to test Ana B. Sánchez, Sergio Segura, Jose A. Parejo and Antonio Ruiz Cortés. Variability Testing](https://reader033.vdocument.in/reader033/viewer/2022050105/5f43727e27ed8b3bb47cabee/html5/thumbnails/44.jpg)
Sprint: Friday
https://www.flickr.com/photos/amazeelabs/9965814443/in/faves-38914559@N03/
Sprint with the Community on Friday. We have tasks for every skillset. Mentors are available for new contributors. An optional Friday morning workshop for first-time sprinters will help you get set up. Follow @drupalmentoring.
![Page 45: SmarTest - isa.us.es · Testing real highly-configurable tools Artificial models to test Ana B. Sánchez, Sergio Segura, Jose A. Parejo and Antonio Ruiz Cortés. Variability Testing](https://reader033.vdocument.in/reader033/viewer/2022050105/5f43727e27ed8b3bb47cabee/html5/thumbnails/45.jpg)