model based testing
DESCRIPTION
Model Based Testing. Aparna Vijaya & Beaulah Vineela P. Supervisor: Antonio Cicchetti. Presentation Outline. Model Driven Development Model Based Testing (MBT) MBT Tools Survey Supporting the automatic test case and oracle generation using system models – Aparna - PowerPoint PPT PresentationTRANSCRIPT
Model Based Testing Aparna Vijaya & Beaulah
Vineela P
Supervisor: Antonio Cicchetti
April 22, 2023 2
Presentation Outline
• Model Driven Development
• Model Based Testing (MBT)
• MBT Tools Survey
• Supporting the automatic test case and oracle generation using system models– Aparna
• Traceability in MBT– Beaulah
April 22, 2023 3
Motivation Significance of Model Driven
Development in Software Engineering- Emerging trend in software industry- Handle system complexity - Deviation from traditional development- Support requirements change- Support for highly distributed systems - Improve consistency - Provide quality solutions - Improve the productivity
April 22, 2023 4
Models and Modeling• What is a Model ?
A Model is an abstract representation of the system• Modeling Languages
- UML - xtUML - SysML
April 22, 2023 5
Generalized approach for MDD
• Shift focus from writing code to developing models
• User works on a platform independent model
• Selects the specific target platform
• Tool generates the code.
“Design once, run anywhere”
www.cs.tufts.edu/comp/180/teams/t4/site/MDD_all2.ppt
April 22, 2023
Model Based Testing – Overview• “Model-based testing is black box testing process
for derivation of test cases from a model that describe functional aspects of the system under test and executing those test cases.”
6
is a partial description of
Model System
Abstract Tests Executable Tests
can be run against
are abstract versions of
can be derived from
April 22, 2023 7 Practical Model Based Testing- A Tools Approach, Mark Utting & Bruno Legeard
• Five Main Steps
1) Model the SUT
2) Generate abstract tests from the model.
3) Concretize the abstract tests to make them executable
4) Execute the tests on the SUT and assign verdicts.
5) Analyze the test results.
Model Based Testing Approach
April 22, 2023 8
Pros & Cons of MBT
• Improved quality of the product
• Better communication using models
• Eases updating of test suites for changed requirements
• Early exposure of ambiguities in specification and design
• Generate variety of test suites from the same model by different test selection criteria.
• Complexity of the models
April 22, 2023 9
Tool Survey Tool Name Type Modeling
LanguageUsability Automatic
generation of test cases
Export models to other formats
Model Simulation
Platform
UPPAAL Academic UML Yes No Can export to XML file
Yes Windows
CowSuite Research UML Yes Yes - Yes Windows
JUMBL Academic TML (Markov Chain Models)
No Yes - No Windows /Unix
Enterprise Architect
Commercial UML Yes Only for Modeling and does not support model based testing
Spec Explorer Research SPEC # and ASML
Yes Yes Can export to XML file
No Windows
Model Junit Academic Java No Yes Can export to XML file
No Windows
April 22, 2023 10
Tool Survey cond … Tool Name Functionality Reliability Usability Efficiency Maintainability Portability Licensing and
pricing
UPPAAL Suitability: supports MBTInteroperability: models nor testcases cannot be interchanged between tools
Fault tolerance: Only if the model is created and simulated without any errors testing is possible. Each phase is completely depended on previous phase
Yes, graphical user interface and drag and drop approach
Time, effort and resource used : less
Analyzability: Yes Changeability: YesTestability: Yes
Installability: easy
Open source and academic tool
CowSuite Suitability: supports MBTInteroperability: models nor test cases cannot be interchanged between tools
Yes, graphical user interface and drag and drop approach
Time, effort and resource used : less
Analyzability: Yes Changeability: YesTestability: Yes
Installability: easy
Open source and academic tool
JUMBL Suitability: supports MBTInteroperability: models nor testcases cannot be interchanged between tools
No, a new language called TTL has to be mastered for modeling
Time, effort and resource used : High
Analyzability: No Changeability: YesTestability: Yes
Installability: time consuming
Open source and academic tool
April 22, 2023 11
Tool Survey cond … Tool Name
Functionality Reliability Usability Efficiency Maintainability Portability Licensing and pricing
Enterprise Architect
Suitability: Supports only modeling and not MBT
Yes, graphical user interface and drag and drop approach
Time, effort and resource used : less
Analyzability: Yes Changeability: YesTestability: Yes
Installability: easy
Commercial tool
Spec Explorer
Interoperability :Databases parts of states, transitions and test suites can be exported to an XML file but not the State machine diagram as whole. So interoperability is not achieved in this tool.
Fault Tolerance: Tool does not support this. Recoverability: If the application crashes it can be recovered as data is stored locally.
Easy to operate with good GUI. But application does not support lower versions Microsoft word 2007.
An efficient MBT tool supporting all phases of MBT – modeling to result comparison
Easy to analyze the application behavior.
Installability: Simple and Easy to install instructions are provided.
Open Source
Model J Unit
There is a provision for user to add his new test generation algorithms.
Recoverability: If the application crashes it can be recovered as data is stored locally.
GUI that is downloaded from internet is not working
Supports all phases of MBT
Since GUI is not working we could not analyze the entire application.
Installability: Easy
Open source
April 22, 2023 12
Problem Statement
“To design a test generator that addresses the whole test design problem from choosing input values and generating sequence of operation calls to generate executable tests that include verdict information”
April 22, 2023 13
Model based testing at Ericsson
• Spirent testing tool
• Qtronic
• Bridgepoint
April 22, 2023 14
Existing Approaches• Combinatorial Design Paradigm
- What is it all about?
- Pros and Cons of this approach
David M. Cohen, Siddhartha R. Dalal, Jesse Parelius, & Gardner C. Patton, "The Combinatorial Design Approach to Automatic Test Generation”
April 22, 2023 15
Existing Approaches contd… • Usecase approach
Clementine Nebut, Franck Fleurey, Yves Le Traon & Jean-marc Jezequel, “Automatic Test Generation: A Use Case Driven Approach”
April 22, 2023 16
Existing Approaches contd… Leirios Test Generator
- Main principles
- Approach in LEIRIOS tool
C. Grandpierre, B. Legeard, F. Peureux , F. Bouquet , “A Test Generation Solution to Automate Software Testing”
April 22, 2023 17
Overview of technologies used in proposed approaches
• Tefkat
• Metamodel
• MOFScript
• TTCN 3
• Ecore model
• XMI format
• State Diagrams
April 22, 2023 18
Proposed Approach (1)• From Class diagrams
Keith Duddy, Anna Gerber, Michael Lawley, Kerry Raymond & Jim Steel, Model Transformation: A declarative, reusable patterns approach
Source Model
(Class Diagram)
Source Model
(Class Diagram)
Tefkat Tefkat
Source Metamodel Source Metamodel
MOF script MOF script
Target Text
(XMI format)
Target Text
(XMI format)
State Diagram State Diagram
Generate Test
(TTCN 3)
Generate Test
(TTCN 3)
Execute tests Execute tests Test Result Test Result
Adaptor
(Ecore model)
Adaptor
(Ecore model)
Traversed transitionsTraversed transitions
Data-flow testing techniquesData-flow testing techniques
April 22, 2023 19
Proposed Approach (2)• From Sequence diagrams
C Doungsa-ard, K Dahal, and A Hossain, “AI Based Framework for Automatic Test Data Generation”
AI Framework
AI framework for determining test path
AI framework for determining test path
Determine coverage for testDetermine coverage for test
Generate tests
(TTCN3)
Generate tests
(TTCN3)
Execute tests in test harness Execute tests in test harness
Test Result Test Result
Sequence Diagram
(Bridgepoint)
Sequence Diagram
(Bridgepoint)
XMI snippet
(Bridgepoint)
XMI snippet
(Bridgepoint)
Control Flow GraphControl Flow Graph
April 22, 2023 20
Analysis & Future work
• Why MOFScript generation in method one.
• Class diagrams with pre and post conditions of the particular state
• Automatic test case generation can be done as a background process while the user is creating models could also be a research area
April 22, 2023 21
Problem statement
“Achieving traceability between the test cases generated and requirements in a model based testing (MBT) scenario”
April 22, 2023 22
Quick glance about traceability • Ensure interrelation between
different artifacts involved software engineering
• Enable to perform quick and easy impact analysis
• Enhance the product quality
Specifications 1. RQ1 2. RQ23. RQ3
DESIGNREQUIREMENTS
FuncX(){Int x;
}
Func y(){Double y;
}
CODE TEST CASES
Test case 1 Test case 2 Test case 3
SDLC Phase - Artifacts
Specification - requirements
Design - design models
Implementation - code( functions, libraries)
Validation - test case
April 22, 2023 23
Requirements Traceability in MBT
Would enable • Relevance of the test case generated from model • How a modification in the model would effect the
generated test case • Provides the effective coverage
April 22, 2023 24
State of the Art for Traceability in MBTRM TOOLS
ReqTracer
DOORS
Requisite Pro DOORS
Requirement Management Tool
Model J Unit DOORKEEPER 4 Eclipse
Model Based Testing Tool
Connector API
April 22, 2023 25
Drawbacks & AlternativeDrawbacks - • Semantic variation between source artifact and target
artifact • Need for an RM tool to support modeling notations of
MBT tool • Integration problems Alternative - • Need to have traceability with in MBT tool • Benefits of traceability through model
transformations with in MDD tools
April 22, 2023 26
Test cases with state chart diagram
April 22, 2023 27
Requirements trace model
• A new requirement trace model is introduced
• Provided as input MBT tool
• All the requirements are represented in trace model and linked to the corresponding system functions
-RQ1-RQ2
Requirements
+alarm on()+alarm off ()+set alarm ()
System Application«traces»
April 22, 2023 28
How this approach works • A transition to new state in state chart diagram is
done by an action or function • New state corresponds to state of system which is
desired to be => in new state system might has fulfilled a requirement
• Requirement identifiers are tagged with corresponding method in trace model
• By generating test cases using state chart diagram and linking trace model, traceability is achieved with MBT tool
April 22, 2023 29
Conclusion – Future work
• This would avoid problems with semantics while linking requirements between different tools
• Not only with state chart diagrams, we can have traceability model linked with other test generation models