using models to automate testing: experiences & challenges · pdf fileusing models to...
Post on 20-Mar-2018
225 Views
Preview:
TRANSCRIPT
Using Models to Automate Testing:
Experiences & Challenges
Dr. Muhammad Uzair KhanSenior Member IEEE
Quest Lab, FAST-NU Islamabad
Uzair.khan@questlab.pk
Pakistan Software Quality Conference, PSQC’17
April 08, 2017
About me!
• Assistant Professor @ FAST NU
– PhD from University of Nice, France and INRIA in 2011
• Head QUEST Lab
• Founding Member, Pakistan Software Testing Board
• Treasurer and Secretary IEEE Islamabad Section
• Over 20 high quality international research publications
• Over 14 years of industry and academia experience
– Expertise: Software Testing , UML, Empirical Software Engineering, Model
refactoring, Aspect Oriented Engineering
• Principle Investigator of ICT R&D Funded project – AGCT 2017 (26
Million PKR)
• Co-Principle Investigator of ICT R&D Funded project – MBT-Toolset
(15 Million PKR)
2
Software Quality Engineering and Testing
(QUEST) Lab
• Quest Lab, Est. June 2013, www.questlab.pk
• Industry-driven research on software quality engineering: testing,
model-driven development
• Targeting real industrial problems
• Several national and international industry partners
• International academic collaborations:
– Interdisciplinary Centre for Security, Reliability and Trust (SnT)
Luxembourg
– Simula Research Laboratory, Norway
• Several local and foreign funded projects
– Worth over Rs. 50 Million
• Research being published at high quality international venues
• Have offered 12+ fully funded MS scholarships
3
Applied Research in Software Engineering
• Useful research must have impact: Research-driven
innovation
• Close industrial involvement
– Conscious effort to understand the problems and practice of
industry
– Focus on pain points in industry
– Realistic Evaluation
– Long term industrial collaboration => impact
• Impact on Academia
– Should contribute to the learning of students
• Introduction of new courses/specializations
– Producing high quality graduates
– Producing high quality publications
4
Our Industry Collaboration Model
• Unlike other stereotyped academic research, at QUEST lab
research is driven by practical problems faced by
industry
• Strong industrial collaborations to develop and evaluate
candidate solutions
5
Software Testing
• Testing is the process of executing a program with the intent of finding
faults
• The challenge is to identify scenarios that trigger the faulty functionality of
software
Major steps of Testing
• Identifying Test Paths/Sequences
• Identifying Test Data
• Test Execution
• Test Evaluation
7
new Calculator(..).add(…)
“Windows”, 2, 2
4
Identifying Test Sequences
• What sequences of function calls to test?
– Out of billions of possible function call sequences for a class, which
call sequences to select?
– Important Decision
– A number of heuristics are typically used, e.g.,
• important function sequences based on application logic
• data-flow sequences
• checking the use-case flows
• checking the misuse-cases
Identifying Test Data
• What test data to execute?
– Consider a program taking two integers as input
• Possible number of input combinations: 264 * 264
• Which combinations to test?
• Decision is typically based on test adequacy criteria
– To maximize code’s control flow coverage
– To maximize code’s data flow coverage
– Test based on equivalence classes and boundaries
Computing Test Oracle
• How to obtain a test oracle (expected results)?
– Computing test oracle is challenging
– Simpler at unit testing level
– Expected outputs are based on test data
– Well documented specifications can help
Test Execution
• How to execute test cases?
• Manual execution
– laborious
• A number of frameworks support automated test execution
• Record and replay tools (e.g., for web applications)
• xUnit framework tools
Automated Test Execution
• How to execute test cases without manual intervention?
• A number of frameworks support automated test execution
• Record and replay tools (e.g., for web applications)
• xUnit framework tools
• Industry practice
– Large part of local industry struggling to achieve automated test
execution..
12
Automated Software Testing – Industry Practice
• Problems?
– Human intensive effort,
– Not scalable, not repeatable, not maintainable
– Limited confidence in testing, How to evaluate effectiveness of
testing?
• Typical solution:
– Testing is driven by availability and capability of the
resource
13
Why not automated other steps?
• Useful Test data, sequences, and oracle are
application specific
– Cannot be generalized
– Need to capture specification of the system
under test in a machine analyzable form
– Domain/application specific methodologies and
strategies for testing are required
• One widely accepted solution: Model-based
testing
– Our expertise at QUEST lab
14
Test Path/Sequence
Test Data
Test Oracle
Model-Based Testing
• Model-based Testing (MBT) uses models of the system to
completely automate the testing process
– The models typically describe some aspects of system under test
• MBT allows automated testing solutions for complex industrial
software systems
– Early
– Systematic
– Scalable
15
Test Case Generation and Execution
16
Test cases
SW Model
SW Code
Test cases
CompareOracle
Expected
Results
Results
What modeling
strategy?
What test data & sequence
generation strategy?
What oracle strategy?
Tailored Modeling
Methodology
Application specific/
control and data flow
analysis
Application
domain specific
Vending Machine Control Software -
State Machine
off
power-on/
light-on; m:=0
power-off/
light-off
not
emptyempty
inc/m:=1
dec[m=1]/m:=0
inc/m:=m+1
dec[m>1]/m:=m-1
money
busyidle
coffee
on
coffee[m>0]/start
after(5)/stop
coffee[m>0]/start;dec
17
State coverage
18
off
power-on/
light-on; m:=0
power-off/
light-off
not
emptyempty
inc/m:=1
dec[m=1]/m:=0
inc/m:=m+1
dec[m>1]/m:=m-1
money
busyidle
coffee
on
coffee[m>0]/start
after(5)/stop
coffee[m>0]/start;dec
Applied Research in MBT
• MBT is not something new!
• Well-accepted by international industry
– IBM, Microsoft, Cisco, Schlumberger, etc
• Next, some of the industrial applications of MBT we at Quest
lab have worked with
19
Automated Software Testing with Models
20
I - Video Game Testing
• Game Industry worth around $100 billion*
– Platform games a famous type of games
• Games are highly interactive, state-based, and are typically real-
time software
• Highly competitive market
21
*https://newzoo.com/insights/articles/
Video Game Testing – Current Practice
• Manual Testing – A common approach of testing games in
industry
• Testers play various game scenarios manually
– Preferably over long periods of time
– Observing various game scenarios
– Various game play rules need to be taken into account for testing
• Focus of functional testing:
– Functional correctness of games
– Identifying crash points
22
Video Game Testing - Problem
• Problem
– Lack of a scalable strategy to test functionality of platform games
• Issues with Current Practices
– Labor intensive: Manual testing by playing games for hours
– Non-systematic: Highly dependent on individual skills
– Boredom effect: Task gets non-rewarding over time
– Non-repeatable: playing the same game scenarios for every
change in the game
– Error prone: A number of interesting scenarios are missed
23
Video Game Testing – Automated Approach
• We developed an automated platform game testing approach
– Focuses on testing the game based on avatar’s behavior by
simulating the inputs required for playing games.
– Testers need to model the game play from users perspective
• We developed a detailed modeling methodology
– A UML profile for modeling platform games for testing
– A set of guidelines for modeling game concepts and behavior for
testing purposes
24
“Automated Model-based Testing Approach for Testing Games”, published in ACM/IEEE 18th
International Conference on Model Driven Engineering Languages and Systems (MODELS’15),
2015, Canada.
Modeling Mario Brothers
25
Video Game Testing – Models for Mario Bros
26
Fig. 3. Mario’s avatar state machine
Fig. 4. Run forward state machine.
V. CASE STUDY
To demonstrate the application of our proposed game
testing methodology, we applied it on two case studies. The
first case study is of an open source variant of a famous
platform game, Super Mario Brothers [9]. The second case
study is of an endless runner platform game being developed
by our local industry partner.
Mario game was first introduced in 1983 for Nintendo
console. Since then, a large number of versions of the game
have been introduced. In Mario brothers’ game, the avatar
moves through various obstacles and collects coins. While
moving, Mario dies when it hits a ghost or falls of the edge.
.
Video Game Testing
• Test Sequences (the user game play events) are automatically
obtained by traversing the models as a graph
• Test Data generated by solving constraints on guards
• Test Execution is automated by developing a simulator that can
execute user events
• Test Oracle
– System-generated events
– State transitions in the state machines (due to a change, signal, or
a time event) that are triggered as a result of the user-generated
events
– System crashes
27
Fig. 3. Mario’s avatar state machine
Fig. 4. Run forward state machine.
V. CASE STUDY
To demonstrate the application of our proposed game
testing methodology, we applied it on two case studies. The
first case study is of an open source variant of a famous
platform game, Super Mario Brothers [9]. The second case
study is of an endless runner platform game being developed
by our local industry partner.
Mario game was first introduced in 1983 for Nintendo
console. Since then, a large number of versions of the game
have been introduced. In Mario brothers’ game, the avatar
moves through various obstacles and collects coins. While
moving, Mario dies when it hits a ghost or falls of the edge.
.
A Sample Bug
28
II - Testing Real time Embedded Software
• Problem:
– RTES execute in complex environments
– A large number of possible environmental scenarios
• Practice:
– Manually writing simulators for SiL testing
– Manually system level test cases
29
Marine Seismic Acquisition SystemBottle Recycling Machine
Testing RTES - Solution
• Independent
– Black-box
• Behavior driven by
environment
– Environment model
• One model for
– Environment simulator
– Test cases and oracles
• Notation: Class Diagrams,
State machines, OCL
30
Environment
Simulator
Test cases
Environment Models
Test oracle
* *Environment Modeling and Simulation for Automated Testing of Soft Real-Time Embedded
Software”, in Software & Systems Modeling Journal (Sosym), 2015
Domain Model - Example
Behavioral Model - Example
32
Automated RTES Testing
• Test Sequence
– Search algorithms guide the selection of test sequences that take
the RTES closer to error states
• Test Data
– Test data generated based on search using constraint solvers
• Test Oracle
– Error states (states that should never happen) were added in the
environment state machine
• Test Execution
– Simulator is automatically generated from the environment models
33
III - Mobile Application Generation & Testing
• Problem
– Mobile applications need to support a large number of platforms
– Very tedious task to develop and maintain numerous versions
– Manually evaluating non-functional properties across various
platform takes significant effort
34
Mobile Application Generation & Testing
• Solution:
– Capture the core business logic with models
• A detailed modeling methodology specific to mobile applications
– UML Profile for modeling Mobile Applications
– UML Class Diagrams and State-machines
– Action Language: ALF
• Native application code generated for each platform
• Performance related properties are modeled and their
conformance automatically tested
35
*A product-line model-driven engineering approach for generating feature-based mobile applications, Journal of Systems
and Software, vol. 123, January 2017
Mobile Application Generation & Testing
Mobile Application Generation & Testing
Android Application Code Windows Application Code
Mobile Application – Modeling Performance
38
Mobile Application – Modeling Performance
39
Mobile Application – Performance Testing
• Test Sequences
– Search-based approaches and fuzzy logic are used to identify test
sequence that stress the SUT to violate the performance properties
• Test Data
– Test data is generated to support the generated test sequences
• Test Oracle
– The performance properties modeled are considered as oracle
• Test Execution
– Tests are automatically executed across various platforms for the
versions of the mobile applications
40
IV - Automotive System Testing
* “Automatic Generation of System Test Cases from Use Case Specifications”, in International Symposium
on Software Testing and Analysis (ISSTA’ 2015), USA
• A safety-critical automotive
software developed
• Monitors a car seat via an electrical
field sensor to classify the occupant
– disables the airbag for children and
unoccupied seats
– ensures airbag deployment for
adults
• Need to support ISO-26262
automotive standard for functional
safety
– All requirements must be tested
with test cases
Automotive System Testing
• Solution: Use case modeling with RUCM and OCL constraints
• RUCM: Restricted Use case Modeling Language
– Allows to specify scenario in a structured language
• Natural Language Processing to generate test cases
• OCL Constraints solving to generate test data
RUCM Use case
Derived OCL Constraints
Automotive System Testing - Automation
• Test Sequences
– Various scenarios of the use cases are automatically identified to
form the test sequence
• Test Data
– The OCL constraints are automatically solved to generate
scenarios to test the system
• Test Oracle
– The system actions of the use case are treated as oracle
• Test Execution
– Test scripts are automatically generated and executed on pre-
existing simulators
45
When to Automate?
• A large number of testing scenarios are automatable
– ROI?
– Large maintenance cycle
– Regression Testing
– Data driven Testing, e.g., web application
– Load and Performance Testing
– When the effort of manual testing is huge and redundant
– Systematic testing is a need
– Systems with complex nature and large environment variations
– Critical systems
• Learning curve should not hinder automation
– Need to start from somewhere
46
What to Automate?
• Everything that can be automated, should be automated?
• A large number of testing tools support automated test
execution
– xUnit framework tools: JUnit, GoogleTest, nUnit, etc
– Record & replay tools: Selenium, Test Complete, QTP, Watir, etc
– Should be used to reduce rework and support systematic test
execution and management
• Load testing is only practical when automated: LoadRunner,
JMeter
• Repeated and labor intensive tasks should be automated, e.g.,
manual white-box testing
• Useful Test data, sequences, and oracle are application specific
– Given the ROI and nature of the project, investment should be
made in exploring such strategies
47
Open Challenges
• How to automate usability testing?
• Test sequence, data and oracle generation approaches are
tailored to the application needs
– Tool support for Test Data Generation is improving
• EvoSuite, SBOS OCL Solver
– Test oracle in most cases is the difficult part
• Model-based testing approaches are promising
– Conformiq, Smartesting
– Still a huge potential for improvement
• Regression Testing approaches have room for a lot of
improvement
48
Future of Software Testing
• Current Software Testing Industry: Over $60 Billion
– 10% growth every year - a few of the industries having a double
figure growth rate for last decade
– Software Testing Job ranked among the top 10 In-demand jobs of
2016 (fortune.com)
• The role of software tester is changing
– More development, Developers are Testers, Test Driven
Development and DevOps, demonstrable skillset to be rewarded
• Cloud and IoT increasing the importance of software testing
• Tools gaining momentum
• Automated testing for tedious and repetitive tasks - A must
• A lot of research prospective!
49
Thanks!
50
Dr. Muhammd Uzair Khan
Quest Lab, FAST house, Islamabad
Uzair.khan@questkab.pk
top related