Faculty of Technologyand Society

Master thesis

Proof-of-concept of Model-based testing based on anUML-model of a water-level measurement system

Zoubida AlshekhlyNamra Jamshaid Gill

Exam: Computer Science, Master’s Programme (One Year)Subject Area: Computer ScienceDate: 08/11-2019

Abstract

Software testing is a very important phase in software development as it minimize risks ina software system, however, it consumes time and can be very expensive. With automatictest case generation time consumption and cost can be reduced. Model-based testing isa method to test a software system with a model of the systems behaviour. Automatictest case generation is often considered a favorable support in model-based testing. In thiswork, the concept of model-based testing is explored along with testing the embedded partof a water-level measurement system (WLM) to investigate the efficiency of model-basedtesting on a software system. As a result of this, a model-based testing tool, MoMut::UMLis used to generate the test-cases on the UML model of WLM system that is built ina UML modeling environment, Eclipse-Papyrus. However, MoMut::UML implements aspecial type of model-based testing technique, model-based mutation testing; that injectsfaults in the UML model, and generates test-data on the fault-based model. By this, thebehaviour of system-under-test, only the UML model of water-level measurement system,is tested.

Keywords: Model-based testing, Model-based testing tool, software system, UML, auto-matic test case generation.

Glossary

LORAWAN: The LoRaWAN specification is a Low Power, Wide Area (LPWA) network-ing protocol designed to wirelessly connect battery operated ‘things’ to the internet inregional, national or global networks, and targets key Internet of Things (IoT) require-ments such as bi-directional communication, end-to-end security, mobility and localizationservices.System under test (SUT):System under test refers to a system that is being tested forcorrect operation.Test oracles:Test oracle, is a mechanism for determining whether a test has passed orfailed.The Object Constraint Language (OCL): is a declarative language describing rulesapplying to Unified Modeling Language (UML) models.

3

Contents

1 Introduction 11.1 Background . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 11.2 Vision of this work . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 21.3 Contribution and novelty of this work . . . . . . . . . . . . . . . . . . . . . 21.4 Motivation of using water-level measurement system . . . . . . . . . . . . . 21.5 Research aim and research questions . . . . . . . . . . . . . . . . . . . . . . 3

1.5.1 Research aim . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 31.5.2 Research question (RQ) . . . . . . . . . . . . . . . . . . . . . . . . . 3

1.6 Scope and limitations . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 31.6.1 Research limitations . . . . . . . . . . . . . . . . . . . . . . . . . . . 3

2 Theoretical background 42.1 Automatic testing on software systems . . . . . . . . . . . . . . . . . . . . . 42.2 Model-Based Testing . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 42.3 UML (Unified Modeling Language) . . . . . . . . . . . . . . . . . . . . . . . 52.4 Model-based testing test criteria . . . . . . . . . . . . . . . . . . . . . . . . 52.5 Model-based testing tools . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6

3 Related work 73.1 Experiences of System-Level Model-Based GUI Testing of an Android Ap-

plication . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 73.2 Model Based Testing of System Requirements using UML Use Case Models 73.3 An Evaluation of Model-Based Testing in Embedded Applications . . . . . . 73.4 Water-level measurement system (WLM) . . . . . . . . . . . . . . . . . . . . 8

4 Method 104.1 Nunamaker and Chen’s methodology . . . . . . . . . . . . . . . . . . . . . . 10

4.1.1 Construction of a Conceptual Framework . . . . . . . . . . . . . . . 104.1.2 Development of a System Architecture . . . . . . . . . . . . . . . . . 114.1.3 Analyze and Design the System . . . . . . . . . . . . . . . . . . . . . 114.1.4 Build the System . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 114.1.5 Observe and Evaluate the System . . . . . . . . . . . . . . . . . . . . 11

5 Implementation 125.1 Problem breakdown . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 125.2 MBT tool literature review . . . . . . . . . . . . . . . . . . . . . . . . . . . 135.3 System Architecture . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 15

5.3.1 System Architecture of whole process . . . . . . . . . . . . . . . . . . 155.4 Model-Based Testing . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 16

5.4.1 UML model . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 165.4.2 Papyrus . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 185.4.3 UML model in Papyrus . . . . . . . . . . . . . . . . . . . . . . . . . 205.4.4 Model based testing tool - MoMuT::UML . . . . . . . . . . . . . . . 22

4

5.5 Perform Model-Based Testing . . . . . . . . . . . . . . . . . . . . . . . . . . 235.5.1 Configuration of MoMut - tool . . . . . . . . . . . . . . . . . . . . . 245.5.2 Running the MoMut - tool with the AlarmSystem demo . . . . . . . 265.5.3 Running the MoMut - tool with the water-level measurement system 29

5.6 Model-based Test suite generation . . . . . . . . . . . . . . . . . . . . . . . 33

6 Results & Analysis 366.1 Analysis of Model-based testing . . . . . . . . . . . . . . . . . . . . . . . . . 36

6.1.1 Challenges with model-based testing tool: MoMut:UML . . . . . . . 366.1.2 Analysis of UML model . . . . . . . . . . . . . . . . . . . . . . . . . 376.1.3 Analysis of MBMT Test-suite generation . . . . . . . . . . . . . . . . 376.1.4 Drawbacks of Model-based testing . . . . . . . . . . . . . . . . . . . 38

6.2 Analyse of the chosen command-line option for MoMut . . . . . . . . . . . . 386.3 Benefits of MoMut::UML for Model-based testing . . . . . . . . . . . . . . . 39

7 Threats to Validity 39

8 Discussion & Conclusion 408.1 Awnsering the research questions . . . . . . . . . . . . . . . . . . . . . . . . 408.2 Future work . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 418.3 Conclusion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 41

References 42

9 Appendix A 45

10 Appendix B 46

5

6

1 Introduction

This section presents the background of this work followed by the vision, the research aimand the research questions. The vision includes the contribution, and the novelty of thework. Moreover, the scope and limitations are also presented.

1.1 Background

Testing is the evaluation process to improve the quality of a product [1]. During theexecution of testing a software system, a failure can be detected. The failures are causedby faults in the software system. The faults are usually caused by human-error in design,code etc. When a failure is detected, the fault can be investigated.There are different types of software testing methods, techniques, and test data genera-tion. The different types of testing differ depending on their scale, characteristics and theinformation needed to design the tests. Examples of testing are; component testing, inte-gration testing, system testing, functional testing, robustness testing, performance testing,usability testing, black-box, white-box testing, and model-based testing which is a form ofblack-box testing [1]. The model-based testing generates functional tests and can be usedfor robustness testing. Figure 1, illustrates a graph of how the different types of testing aredivided into three axis. Every axis represent an aspect in testing, the first axis representsthe scale of the system under test, SUT, the second one represents which characteristics totest, and the third axis shows the type of information to test. The figure also shows howthe model-based testing is related to these different axis.

Figure 1: Different types of testing

One of the fundamental factors in software testing is the automatic test case generation.Automatic Test case generation for software testing indicates a process that generates testcases automatically. The automatic test case generation technique is convincing as it re-duces the cost and time, and increase the system efficiency, when it is compared to other

1

test generation processes [2].

The model-based testing (MBT) technique, is one of the popular searched phrases in con-nection with automatic test case generation. Model-based testing is built with the modelof a system, in order to generate test cases of the model of a system [3]. There are multipletypes of models that can be used in MBT. However, the most practical and widely usedmodel for supporting the MBT technique is the unified modeling language, UML[4]. MBTtechnique is used in tools that are able to generate test cases by analyzing the behavior ofthe UML model. Model-based testing have different test criteria, and one of these criteriais used in this work and mentioned in section 2.4 [1].

The SUT used in this work is a water level measurement system [5]. The system consistsof different technical parts: embedded, dashboard as a frontend, and a network part. TheMBT is performed on the embedded part of the system. The motivation for selecting theembedded system part is mentioned in section 1.4.

1.2 Vision of this work

The vision of this work is to provide a proof-of-concept of MBT. For this aim MBT isperformed on an embedded system of water-level measurement system which is presentedin section 3.4. Subsequently, a guide on how to implement and perform MBT is providedbased on the experience developed while performing MBT.

1.3 Contribution and novelty of this work

The contribution of the work is to provide the IT industry a base to start adopting MBTby creating an easy-to-follow guide on how to implement MBT. The guide indicates a clearexplanation on how to test an embedded system using a specific MBT tool. In this way, aproof-of-concept of the MBT is established to illuminate the privileges of MBT approach.

1.4 Motivation of using water-level measurement system

The motivation for using the water-level measurement system, WLM, as a test material forthis work is because the system is already manually tested [5]. The automatically generatedtest cases are conducted by using a MBT tool. The reason behind selecting the embeddedsystem part and not the other associated parts; the front-end part is presented as an opensource dashboard, and the interface of the dashboard is not reachable for testing. Thenetwork part of the water-level measurement i.e. LoRaWAN network is also unreachablefor executing test cases through MBT. Although, the embedded part of the system worksas a state-machine and therefor is reasonable for MBT to test.

2

1.5 Research aim and research questions

1.5.1 Research aim

The aim of this work is to provide a guide that explains how to generate test cases alongwith a model-based testing tool. To clarify, this work is a proof-of-concept of model-basedtesting implemented on a embedded system with test cases generated based on the UMLmodel.

1.5.2 Research question (RQ)

RQ1 What are the steps that should be followed to perform Model-based testing with aUML-model?

RQ2 How to use the chosen Model-based testing tool to generate testcases by using anUML-model of an embedded system?

RQ3 How to build an UML-model of the embedded system of water-level measurementsystem?

RQ4 Which type of Model-based testing technique is used by the chosen Model-basedtesting tool?

1.6 Scope and limitations

This work has limitations as the scope of MBT is broad. The research limitations arepresented in the following section.

1.6.1 Research limitations

The guide on how to perform MBT with a MBT tool is done by using specifically anUML Model. Furthermore, as mentioned, the proof-of-concept will only be implied on anembedded system of WLM system. A literature review about the available model-basedtools is conducted in order to choose the MBT tool for performing MBT on an embeddedsystem. There are different type of MBT techniques presented, however, only one of themwill be explored in this work.

3

2 Theoretical background

This chapter presents the theoretical background about the relevant topics of this work.

2.1 Automatic testing on software systems

Testing is an essential part for a software development of software system. Testing pro-vide the quality assurance of the software solution. Manual testing are usually performedon the software system solutions. Thus, as the complexity and the size of the softwaresystem grow, further more time and labor-force are required. To avoid such labor-forceand minimize the time-consumption, automatic test case generation can be performed [6].There are many reason of why test automation is beneficial compared the manual testing.According to the research analysis [7], performing automatic testing in software systemslead to: improving product quality, achieving high test coverage of code, reducing testingtime, allowing the re-usability of tests, reducing cost and increasing fault detection. Per-forming automatic testing on a software system helps to provoke failures in the system[8]. By provoking faults in the system, the information about the corresponding faults areconveyed and the failures can be dealt with. The software testing process consists of threeparts: test case generation, test execution, and test evaluation [6]. There are several typesof automatic test case generation. The automatic test case generation approaches [9] are:specification based test case generation, model-based test case generation, path orientedtest case generation and intelligent technique.

2.2 Model-Based Testing

Model-based testing (MBT) is a growing software testing technique that has many ben-eficial earnings, such as; initial specification and error detection, improving test docu-mentation and communication between the developer and testers, and lower cost[10]. Asmentioned earlier in section 1.1, the model-based testing is a form of black-box testing,because the test cases are generated from a model that is obtained from of the behavior ofthe system under test (SUT) or its environment. MBT tests are based on the model whichin turn is used by a MBT-tool that generates the test cases from it [1].

Model-based testing have four different approaches:

• "Generation of test input data from a domain model."

• "Generation of test cases from an environment model."

• "Generations of test cases with oracles from a behavior model."

• "Generations of test scripts from abstract tests." [1]

The most used MBT approach is the third one, it focus on the SUT behavior such as; theexpected output values of it. It generates test cases, as well as, generation of oracles thathave the task to check the output of the SUT. This kind of testing is very efficient, because

4

the sequence of the generated test cases can be transformed into test scripts directly.

The use of MBT can vary dependent on the system or the tool utilized. Further advantagesof MBT that are good to mention; its potential to evaluate SUT quality and millions oftest suites can be generated automatically [11].

2.3 UML (Unified Modeling Language)

UML is generally used as a model support for MBT. There are many advantageous rea-sons for using UML, such as; supplying a broad collection of diagrammatic notations, theObject Constraint Language (OCL) combined with UML in order to utilize more accuratemodels. In addition, UML is a standard for modeling, which make it qualified in order toprocess a MBT. UML consists of a set of diagrams [12]. Some of these diagrams can beused in MBT tool.

The UML diagrams that can be used for MBT are: class diagram, object diagram, andstate machine diagram. These diagrams make a base for MBT, every diagram has a task inautomatic test generation. The class diagram models the control points and SUT observa-tion, while object diagram defines the test data. The state machine and OCL collaborateto model the dynamical behavior of SUT [12]. The process of conducting an UML modelthat is appropriate to MBT, is by adding the details about the behaviour of the methods ina class diagram. The reason for that is the state-machine diagram does not give sufficientinformation about the behaviour of the SUT [1].

2.4 Model-based testing test criteria

There are different kinds of test selection criteria. These criteria determine the ability ofMBT tool to control the selection of tests. The test criteria selection process depends onthe data coverage of the SUT that leads to good test suites. The test criteria is basedon the model and the requirement of the SUT, and not on the source code of the model.There are several types of test criteria such as data coverage criteria, fault-model criteria,requirements-based criteria etc. [1].

Data coverage criteria creates test data on specified data input values, this can beuseful when there is huge number of possible input values.

Fault-based testing is a software testing technique that is done by employing the faultthat occur very often. The idea of fault-based technique is to use fault models to generatetest data.

Requirements-based criteria is a software testing technique that generates test-data onthe functional requirements of the SUT. This is useful to verify whether the requirementsof a product is fulfilled [1].

5

2.5 Model-based testing tools

Model-based testing tool generates test cases which can later be automatically deployed bya computer [11]. Several Model-based testing tools are available today, both commercialtools and prototypes. In order to decide which MBT approach suits best for a softwaresystem, it is important to understand which type of techniques and tools to fulfill thesystems test requirements. A categorization is applied for the possible approaches [1]; thecategorization is divided into following: determining the subject of the model, reflectingon the source of the test model, determining the characteristics of the model, determiningthe style and notation of the model, reflect on how the tool approach supports test criteriasuch as data coverage criteria, requirements-based criteria etc, and whether it should beonline or offline testing [1].

6

3 Related work

This chapter presents the related works. All of the papers that are presented in this sectionhave similarities and differences with the research in this work.

3.1 Experiences of System-Level Model-Based GUI Testing of an An-droid Application

In the IT-industry, companies hesitate to use MBT technique, because there is a lack ofeasy to use tools for MBT, and difficulties for finding new test model experts. However, T.Tommi, et al. [13], is aiming to find a solution for finding an user-friendly tool by develop-ing a new MBT tool used for Android system on mobile devices. The developed tool-setis called TEMA tools and it is a commercial tool. The Tema tools consist of further toolsfor the different aspect of MBT which are; test modeling, test design, test generation andtest debugging. The SUT in Tema tools functions as a state machine. The result of Tematoolset is considerable, the modeling process is effective for finding errors. Despite the bugsfound in Tema tools, it can anyway provide a maintainable regression testing when a newrelease is out. Additionally, the tool helps out with reducing the number of test cases byediting the model with a few steps.

The comparison between the related work and this work, is that both are aiming to useMBT to generate test cases by using a state machine. However, a new tool is developedin the related work. In addition to, T. Tommi, et al. are testing an Android application.

3.2 Model Based Testing of System Requirements using UML Use CaseModels

B. Hasling et al. [14], uses a MBT tool provided by Siemens for a medical solution. Thecontribution is to utilize a use case model as a basis for MBT of system tests, which meanscombining the requirement process with a testing process by use case model. The ad-vantages of using the use case model are; reviewing models, verification and validation,improved requirements, improved test coverage, tracing to requirements and early start inthe development life cycle. The tool used called TDE/UML and it is developed by Siemens.The testing team found it useful to apply MBT using use case model.

The contribution of this related work is to test only the system requirements and not thesystem, which make it different from this work that is testing an UML model of an em-bedded system. The other differences are B. Hasling et al. are using UML use case model,and using a TDE/UML MBT-tool for generating test cases.

3.3 An Evaluation of Model-Based Testing in Embedded Applications

S. Weißleder et al. [15] performed MBT on an embedded system to investigate the advan-tages of automatic MBT generation. The UMLmodel is used for designing the functionality

7

of the system. It assumes that MBT generation process reducing 85% of costs for test de-sign. However, the aim is to prove if the value is true and if MBT is so powerful. Theauthors explain that the testers in the organization need to be trained into MBT, andchanging the workflow from writing test cases to design models instead. In addition, theinitial cost of MBT framework and tools can be a barrier. However, the exchange froma regular testing method to MBT pays off all the costs. Even though, the managers arenot convinced about this and not willing for test method exchanging. The aim is reached,and MBT is an efficient testing tool as promising, although, the transformation process toMBT can be useful only for a similar case study and not for others.

The differences between this work and the related work are; it is not clear which tool isused for MBT and aims to check the efficiency of MBT. The similarities are; an embeddedsystem is tested in both works and an UML state machine is used to present the SUT. Inthis related work, the researcher claims that MBT reduces test costs which is not evaluatedin this work.

3.4 Water-level measurement system (WLM)

The model-based testing, MBT, is implemented on an embedded system which is a partof the water level measurement system [5]. The water level measurement system includesdifferent parts, which are; network, dashboard as a frontend part and hardware. The mainpart of the entire system is hardware. The water level measurement is placed in the wellsusing a sensor connected to a microcontroller. The communication link between differentparts is LoRaWAN, which may either be built into a microcontroller or be built separately.The rest of the hardware part for the product consists of an encapsulation box and abattery. The system is already tested manually in the lab environment, and the aim ofthis work is to implement an automatic test method on this system. An overview of thewhole system including all the parts is shown in figure 2.

8

Figure 2: Water level measurement system.

9

4 Method

This section explains how the research work is conducted and which research methodologyis used in order to answer the research questions.

4.1 Nunamaker and Chen’s methodology

The research process of Nunamaker and Chen’s research methodology consists of five stages[16] [17]. See figure 3 below. Each stage has its own objective. If new knowledge abouta specific domain is acquired by researchers, changes in already made design decisions inforegoing stages occur. The change made in foregoing stages improves the quality of thesystem and the research results [17]. It is why, this research methodology is “of an iterativenature” [16]. This research methodology is used in this work due to its iterative nature. AsMBT is performed by analyzing the MBT tool and experimenting; going back and forthin the stages is helpful.

Figure 3: Overview of Nunamaker and Chen’s research methodology.

4.1.1 Construction of a Conceptual Framework

This is the first stage in the research process, to formulate a research question. SinceNunamaker and Chen’s research methodology is used to create the proof-of-concept, a

10

clear picture of what is already presented in the literature has to be understood. This isdone through literature review. This is done before any formulation of a research question.The formulation together with the research aim is in section 1.3. The formulation is donethrough daily meetings and discussions with the supervisor. Also, the chosen MBT toolis selected through the literature review. Several MBT tools in literature and physicallyavailable are studied; in order to find a MBT tool that is most optimal to achieve the aimsof this work.

Moreover, the research aim is divided into sub-goals, also called sub-problems. There-after solution for each sub-problem is allocated and presented as a problem-tree. Theresult of this stage can be seen in section 5.1.

4.1.2 Development of a System Architecture

The second stage in the research process is developing a system architecture, and it’s doneby creating an architecture of the proof-of-concept of this work. All the associated partsare collected to make an overall system architecture such as: water-level measurement,and the MBT tool. The architecture is done by studying how the parts are related to eachother and what functionality they provide. The system architecture provides an overviewof all the different parts needed in MBT. The result of this stage is presented in section5.3.

4.1.3 Analyze and Design the System

During the third stage, the UML model of the system is designed. The model is designedby studying the embedded system functionality of the WLM system. The designed UMLmodel in this stage is build in the next stage with the chosen UML modeling environment.Moreover, the functionality of the chosen MBT tool is analyzed. The result of this stageis presented in section 5.4.

4.1.4 Build the System

In the fourth stage, MBT is performed with the chosen MBT tool. Firstly, the UMLmodel is build and verified with the chosen UML modeling environment. Secondly, theUML model is connected to the MBT tool. Thirdly, MBT tool is executed to generate testcases. While implementing MBT, an easy-to-follow steb-by-step guide is illustrated in thisstage. The result of this stage is presented in section 5.5.

4.1.5 Observe and Evaluate the System

In the last stage of Nunamaker and Chen’s research methodology, the result of the gen-erated test cases by MBT tool is presented. An evaluation of MBT approach is donedepending on the result of the automatic generated test cases from the MBT-tool. Seesection 5.6 for the results of this stage.

11

5 Implementation

This section presents implementation of this work in order to create a proof-of-concept ofMBT.

5.1 Problem breakdown

The proof-of-concept of MBT is divided into three sub-problems. In order to solve thesub-problems, solutions are allocated in some steps for each main sub-problem:

• Choosing a MBT tool

A relevant list of MBT tools is chosen. The MBT tools presented in the list are delimitedby their type and input format. Only tools that have UML models as input are relevant forthis work. When the literature review is done as presented in section 5.2, an experimentalstudy is conducted on the MBT tools that are likely to be the most relevant to test UMLmodel. As the result of this, one MBT tool is selected.

• Creating an UML model?

In order to create the UML model, the states and the state transitions are analyzed ofWLM system in section 5.4.1. The UML state machine is a simple state machine withonly 3 states, LOW, MEDIMUM and HIGH. The UML model is created and verified in anUML modeling environment in section 5.4.2 to use the model for MBT test case generation.

• Creating a user-friendly guide on Model-based testing?

In order to create a guide, every step done while performing MBT is illustrated in theguide. Moreover, the guide includes risks and difficulties faced while performing the test-ing. The guide is presented in a step-by-step form as a user manual.

A complete problem-tree is created as presented in figure 4 below.

12

Figure 4: Problem-tree: problem breakdown with the solutions

5.2 MBT tool literature review

In this work, a list [18] of MBT tools is investigated. The MBT tools in the list arecategorized on the basis of input-format and the type of how they are accessible. Therelevant and available MBT tools from the mentioned list are presented in table 1 below.

13

Table 1: Relevant MBT tools and Modeling Environment

Tool Input format Type DescriptionConformiqDesigner

UML StateMachines,QML

Commercial Models can be created as UML StateMachines and in Qtronic ModelingLanguage (QML). Tests can be ex-ported to test management tools orTTCN-3.

MoMuT::UML UML state ma-chines, OOAS

Academic MoMuT is a family of automated,model-based test case generationtools that can work off UML StateMachines, Assume-Guarantee Con-tracts (REQS), and Object OrientedAction Systems (OOAS). The toolsfeature a fault-based test case gen-eration strategy.

RT-Tester UML/SysML,Matlab

Commercial RT-Tester starts from UML/SysMLor Matlab models, transforms themto a internal representation basedon Kripke structures, transform re-quirements to LTL formulae, andgenerates test cases using an SMTsolver based on the goals from re-quirements and various model cov-erage criteria.

SmartestingCertifyIt

UML / BPMN+ OCL

Commercial CertifyIt is a complete MBT tool us-ing behavioral models (in UML /BPMN with OCL constraints) anda large set of coverage criteria (re-quirements coverage, model struc-tural coverage, pattern-based cover-age).

Three of the MBT tools, are commercial types as shown in table 1, and therefor not pos-sible to test. Conformiq Designer requires that in order to test their product the projectshould be associated to a company. Moreover, a request with a detailed purpose needs tobe sent to the owner so they can approve the request and make the MBT tool available.Likewise, the MBT tool, RT-Tester costs and is not available immediately either. More-over, Smarttesting CertifyIt and MBTSuite [19] are not available as well. However, MBTtool, MomuT::UML is available on the site as an open source software. Aside from the listof MBT-tools mentioned above, the modeling environment PolarSys [20] is tested besidesthe chosen one, Papyrus. Papyrus is selected, it’s easy to use in compare with Polarsysbecause the user manual is present for Papyrus. The steps for implementing the UMLmodel are illustrated in section 5.4.2. As the result of this, MoMuT::UML is chosen as theMBT tool to perform MBT, and Papyrus as a modeling environment because it is required

14

for MoMut::UML.

5.3 System Architecture

In this section, the whole process of this work is illustrated as a system architecture,with the association between the different elements; WLM system, UML model, modelingenvironment and MBT tool.

5.3.1 System Architecture of whole process

The abstract view of how all the parts in the MBT process are connected is shown in thesystem architecture presented below.

Figure 5: System Architecture of Model-Based testing

The UML model of WLM system is the SUT used for MBT. The UML model focuses onthe behavior of the WLM embedded system. The MBT process starts by implementingthe UML model in the UML modeling environment Eclipse Papyrus, and the UML modelis thereafter verified by it. Thereafter, a "bridge" between the UML model and the MBT

15

testing tool MoMut::UML is built. The "bridge" is built through an UML profile1 called tcgprofile, which is described in section 5.5. When the MBT tool is executed, it generates testcases based on the Model-based Mutation testing technology, which is presented in section5.4.4, in which the mutated UML model is converted into object-oriented action systemlanguage OOAS2, and two distinct test case generation back ends are used; EnumerativeTCG, that runs an explicit state exploration of the SUT, and Symbolic TCG that usessymbolic techniques3. Moreover, an external back-end is used to handle more complexmodels called Z34. The backends are applied to the OOAS in order to generate test-caseson SUT; UML model of WLM embedded system. The whole process is illustrated in detailsas a guide in section 5.5.

5.4 Model-Based Testing

In this section, the elements used for MBT are presented.

5.4.1 UML model

System architecture of SUT, WLM system

The architecture shown in figure 6 demonstrates the embedded part of the WLM systemand the functionality of it. The hardware components used are; battery, encapsulationbox, microcontroller with LoRa connectivity and a sensor. The hardware used to givefunctionality to the system; the sensor measures the data, and the microcontroller readsthe data from the sensor. Consequently, the data is sent from the microcontroller to theLoRa gateway[5].

1UML profile is a lightweight extension mechanism to the UML standard2A formalism well-suited for expressing object-oriented models of embedded systems.3The symbolic method is a technique for counting combinatorial objects. It uses the internal structure

of the objects to derive formulas for their generating functions.4Z3 is a theorem prover from Microsoft Research.

16

Figure 6: Architecture of the embedded system for the water level measurement system.

Depending on the measured values from the sensor, WLM is divided in three differentlevels: Low, Medium and High [5]. The logic of how the system works is illustrated inAppendix A as an activity diagram.

UML state machine

As mentioned in the previous section, the state machine of WLM system only consists ofthree states; LOW, MEDIUM, and HIGH. Therefore, the UML model of the WLM systemis a basic UML state machine model with three states. The initial state of the SUT is setto LOW. The trigger for each event is the measured level by the sensor, depending on thewater-level the transitions in the state machine are made, see figure 7 below.

17

Figure 7: State machine diagram of WLM system

As seen in the figure above, every state is reached when a certain percentage of water-levelis measured. If the water-level is between 0-50% of the maxiumum height of e.g a well,the state is set to LOW. If the water-level exceeds to 51-75% the state is transitioned toMEDIUM, and if the water-level is measured higher than 75%, the state is transitioned toHIGH.

5.4.2 Papyrus

In order to perform MBT with MoMut::UML on WLM system, the UML model of WLMsystem is build in an UML modeling environment, Papyrus. Papyrus is an UML modelingenvironment built on the extensible Eclipse framework. It is possible to view differentaspects of a system by using several diagrams in Papyrus. The diagrams are associatedwith the model of the system in which all modeling elements are placed. The modelingelements are used in the diagrams. It is also possible to adapt UML to specific domainsby creating UML profiles in Papyrus.

The installation of Papyrus is done in several steps, firstly, the Eclipse standard has to beinstalled, thereafter Papyrus is installed on top of Eclipse Standard as an extension.

18

Three types of resources are created when modeling with Papyrus as shown in the figurebelow:

Figure 8: UML model resources in Papyrus

• .di file contains the status of the workbench, which diagrams and views are opened,etc.

• .notation file persists the information about the diagrams in the model.

• .uml file persists the UML model.

Therefore, the model is contained in three files. The diagrams are created in the ModelExplorer which is placed in .di file, where a diagram editor together with its tool paletteand outline view are also opened.

To model the WLM system, diagrams such as Class diagram and State machine diagramare used. The Class diagram illustrates the static view of the system and is also usedto specify the inheritance structure in the system. The State machine diagram is usedto specify the behavior of the class. Both of these diagrams are combined in order toimplement MBT on the behavior of the SUT, WLM system. The main elements in thestate-machine diagram are:

• States

• Transitions, lies between states

• Effect ; that specify detailed behavior to be executed when an associated transitionis taken

• Triggering events; which defines the event that makes the transition to be taken

• Pseudo states, e.g. initial, final and choice points

• Reception, designates a signal and specifies the expected behavioral response.

More detailed information about Papyrus can be read in the Papyrus Starter Guide [21].

19

5.4.3 UML model in Papyrus

The UML model of WLM system consists of three diagrams in Papyrus. Two class diagramsand one state machine diagram. Each diagram has a purpose. The class diagrams are usedto illustrate the static view of the system and are also used to specify the inheritance struc-ture in the system. The state machine diagram is used to define the behavior of the system.

To run the MBT tool with Papyrus, instances are needed; which are the objects in theUML model. The first class diagram with instances specifies the two objects and theirlinks in the UML model of WLM system [22]. The diagram presented below in figure 9illustrates the instances of the WLM UML model. A controller instance is an object of typelevel-classifying controller, and an outside instance is an object of type environment. Theassociation between the controller and outside instance is made so the controller knowswhere to send its output data.

Figure 9: Instances of WLM system

The second class diagram defines the context of the WLM UML model as presented infigure 10. This is the test interface which shows what the UML model can receive andsend out.

Figure 10: Context of WLM UML model

20

There are two signals defined for the UML model as seen in figure 10. The signal;level_measured, is shown on the left of the figure and the signal; level_range, is shown onthe right of the figure. The level_measured represents a signal with data that is measuredwith the sensor in WLM system; an integer value which is defined with upper_boundand lower_bound values, this data is the input to the system. Level_range represents asignal with level determined by the controller of SUT; enumeration defined 3 values; low,medium, high. The controller decides the level depending on the measured data; the levelis output of SUT.

The SUT is the UML model of WLM system, the main controller of SUT is defined aslevel_ classifying_controller. Level_classifying_controller has a reception that receivesthe signal level_measured. Level_classifying_controller has to be marked as active inPapyrus so it can continuously react to input; the classes that has double lines on thesides are marked as active in Papyrus. As shown in figure 10, the environment also hasa reception and the reception is for signal level_range, the expectation is that the SUTknows its environment because there is an association to access it. The stereotypes “sys-tem_under_test”, and “environment” are tool specific and used by the tool to determinethe test-interface. The MoMut::UML engine uses UML model in papyrus to generate test-cases. There are two terms used in MoMut::UML to adress the input and the output of theUML model; the controllables are the input to the SUT, and the observables are outputof SUT [23].

Figure 11: State-machine diagram in Papyrus

The state machine presented in figure 11 is divided into two regions, so that each region canprogress individually. The left region has only one transition, the signal level_measuredis assigned to it. The transition has a signal trigger that is triggered by signal events.The transition has no guards, so when the signal is received an effect is created. As ofnow, MoMut only supports effect in a proprietary language which is OCL. In the second

21

region, when a transition is made into one of the states, HIGH, MEDIUM, LOW, a signallevel_range is send out to the environment with the parameter low, medium, high, whichare all entry actions. The action is taken each time the state is entered by one of transitions.There are four transitions, they have a guard which only compares the internal value, themeasured_level with the triggered values that are defined.

5.4.4 Model based testing tool - MoMuT::UML

MoMuT::UML is a MBT tool that functions as a black-box testing [24]. It uses fault-basedtesting methodology; the tool generates test suites that detects models that contain certainselective and seeded faults. The faulty models are also called mutants.

MoMuT::UML uses UML models such as UML state machine diagram, class diagrams,and instance diagrams as input. In this work, the tool converts the UML model to an ob-ject oriented action system (OOAS), which is the interpretation of the UML model. Themutated models of the UML model are created through applying a set of user-selectablemutation operators 5. These mutated models are then converted to OOAS as well. Whenthe OOAS version of the original model and mutated models is presented, MoMuT::UML isready for test suite generation. The generated test suite have the same level of abstractionas the UML model[24] [23].

Model-based Mutation testingThe model-based mutation testing(MBMT) is a fault-based technique and functions asfault-based criteria. It’s a way of presenting the inconsistency of the original specificationand the faults of the SUT [24]. The MBMT process is proceed through placing mutation-operators to the original model. Mutations-operators are placed to the original model indifferent model locations, and in this way a mutant6 is obtained. The behavior of mutatedmodel and original model is compared and as a result of this, test data is generated [22].

Process of Momut::UML

When the Momut tool is executed, the UML model is translated into internal representa-tion called object-oriented action system OOAS. When translation is done, the model istransformed to LLVM7 intermediate representation and executed by the embedded partof MoMut::UML. The LLVM is used to convert the model into machine code so the Mo-Mut::UML tool can use the model in code for exploration in test case generation process.Thereafter, MoMut::UML takes the original model, and applies one mutation operator atone particular location at a time, then a mutant is created; the mutants are also convertedto OOAS. Consequently, the behavior of the original model with the model that have mu-tant is compared, and once a difference is found (observed), MoMut writes out a test case.

5 A simple syntactic change in the code to generate a mutant6Faulty specification7The LLVM compiler infrastructure project is a set of compiler and toolchain technologies, which can

be used to develop a front end for any programming language and a back end for any instruction setarchitecture.

22

The test case describes the location of the mutant in the UML model. The tool is startedthrough the command prompt, and the test case generation process begins. The result ispresented in form of html file that demonstrate the generated test cases besides a list ofmutants operators that tells what changes are made in the model by the replacement ofthe operators. In addition, a number of mutants kill ratio is presented, a mutant is killedwhen the behavior in the model is observable. Observable is a term used in MoMut thatindicates the detected change in the behavior of SUT when a mutant-operator is applied,i.e. the output of SUT, for WLM system UML model the observables are the levels; LOW,MEDIUM, HIGH [23].

Mutants

As mentioned in the previous section, a mutant is killed when a behavior of the mu-tated model is changed. There are two types of kills, weak kill and strong kill. A weak killoccurs after a specific location in the model is visited by a mutant, then the inner state willdiffer from the original. Strong kill is when the test can find the mutant by studying theoutput of the SUT. Equivalent mutants are the changes that do not affect anything on themodel, and are not telling the quality of the tests by not presenting the fault that could befound in a test. On the other hand, there is non-equivalent mutants that actually affect thebehavior of the test and ensures the quality of the test. Moreover, undiscovered mutantsare mutant operators that were placed on the UML model but could not be discovered byMoMut::UML. The test report presents the failures as well, they are the cases when thesystem goes to endless loop or its stuck in one state. In every generated test-suite, thereis a statistic that demonstrates how many mutants a test-suite covers [23].

Command-Line options of Momut

Momut::UML tool have a list of command-line options. The purpose of using command-line options is that every option determines a certain function when generating test cases,i.e. different options provide different test cases. In this way, every SUT should have aspecific option that is related to the system specification. By choosing a suitable option,relevant test cases are generated for the SUT. The right option for SUT is dependant on thesystem specification and the grade of test quality required. A list of options is presentedin Appendix B. In this work, one of the command-line options is used; -strategy, whichis the method used to define the strategy of test-case generation by Momut. There aredifferent strategies, for instance, Random, RRT, BFS, etc. In this work, the strategy BFS(Bredth First Search) 8 is used.

5.5 Perform Model-Based Testing

In this stage, a step-by-step guide is illustrated by using MoMut-tool.8Breadth first Search is a recursive algorithm for searching all the vertices of a graph or tree data

structure.

23

5.5.1 Configuration of MoMut - tool

To configure the MoMut tool, the following steps need to be followed:

1. Download the MoMut tool [25]. The version used in this work is the latest release in2018 as presented in figure 12.

Figure 12: Download MoMut

2. Extract the files from the downloaded compressed file. After extracting the files oneshould see the displayed files in figure 13. The Jar file is the tool engine that includes allthe logic for the MoMut tool to be runnable.

24

Figure 13: Jar file

25

3. Choose the properties of the Jar file and unblock it, as shown in figure 14:

Figure 14: Unblock the jar file

5.5.2 Running the MoMut - tool with the AlarmSystem demo

To run the tool with the associated UML model there are steps to follow. Firstly, thetool is started by a demo which is AlarmSystem. The purpose for running a demo first,is to figure out how the tool works and how the Tcg-profile (Test case generator) projectis connected to the SUT. AlarmSystem demo is chosen because it is less complex thanother demo-projects. The Tcg-profile is an UML-profile that is presented in Tcg-profileproject. This UML-profile needs to be added to any SUT when testing with MoMut, as itis a condition for MoMut::UML to work [22]. The demo projects and Tcg-profile projectare provided with the tool when extracting the compressed file as seen in figure 13. Tcg-project that includes Tcg-profile is imported because it creates the bridge between theUML model in Papyrus and the Momut tool by using the stereotypes: system_under_testand environment, as shown in figure 15.

26

Figure 15: Stereoptypes and environment

1. AlarmSystem is imported as an existing project in the workspace to the Papyrus alongwith Tcg-project to view the UML-model of both projects.

Figure 16: AlarmSystem

2. Modify the runTCG.cmd file by chosing the operations that user want MoMut toperform. The operations chosen in AlarmSystem are "mode" and "steps" as shown infigure 17.

Figure 17: Change or choose operations in the runTCG.cmd file

3. By running the runTCG.cmd file after saving the changes made in the step before,MoMut is started.

27

Figure 18: Running the AlarmSystem

4. After the running process is done, an output file is generated by the tool and in therea report for the test cases is presented as a HTML file.

Figure 19: Test cases output

5. Test-suites presented below are generated, the test-suites are for the Alarm-system’sUML model and therefore irrelevant to analyse in this work. The details in the test-suitesof the UML-model of WLM system are explained in section 5.6.

28

Figure 20: Test cases output

5.5.3 Running the MoMut - tool with the water-level measurement system

In this section, steps of running the WLM UML model with Momut-tool are presented.

1. Create a file with .cmd extension to connect the tool and the UML model. By creatinga .cmd file, it makes the project runnable in association to the tool just by mouse doubleclick.

Figure 21: Run Test case generation file

2. In the .cmd file we need to add the path of the .jar file (MoMut-engine) location on thePC, and referering to the UML in the project and choosing the operations needed.

29

Figure 22: Editing the commandline file

3. Before running the .cmd file, a comparison is made between the UML-model of Alarmssystem and the UML-model of WLM system while creating the UML-model in Papyrus.The comparison is made to explore how the diagrams are connected to the tool. A com-parison is done for the UML elements that are used in Alarm System UML model, whichcan be useful for WLM UML model as well. The comparison also helps to understandPapyrus configuration for a state-machine UML model. There is no guide that explainsthe steps in details on how to prepare UML model in Papyrus to use in MoMut::UML. Allthe steps presented in this guide is based on comparison with AlarmSystem, experimentingstep-by-step, correction of errors and some guidance from MoMut::UML creators [23].

4. The first thing to compare is the structure of the AlarmSystem in the Model-explorer.

Figure 23: Model explorer of AlarmSystem

5. According to the AlarmSystem structure an UML class-diagram is created and thestate-machine is associated in the class diagram as a nested class. In this way, a classdiagram is created that corresponds WLM system and illustrated in figure 10.

6. To connect the UML model of WLM system to TCG-profile we needed to add the path

30

of TCG-profile in the class diagram, by clicking in empty place in the WLM diagram andchoose profile and then add tcg.uml from the list.

Figure 24: Adding tcg.uml path to WLM

Figure 25: Adding tcg.uml path to WLM

7. The system_under_test and environment stereotypes are added to the classes by im-porting the required stereotypes from TCG-project and adding to the profile of each class.

31

Figure 26: Adding stereotypes from tcg

8. The state machine is added to the controller of the SUT as a nested classifier, by rightclicking the SUT and adding state-machine as a "New Child".

Figure 27: State machine as a nested class

9. A signal is created with a signal event and the signal event has a trigger, the signalrepresents the measured level. A reception is created as well, and it has the function of

32

receiving the triggered signal from the level_classifying_controller.

Figure 28: Signal trigger event

10. When the UML model is connected to the tool, it’s time to run the tool for generatingthe test cases. Choose the .cmd file created in the first step of this section for runningprocess.

5.6 Model-based Test suite generation

In this section, the test case generation of MoMut:UML is described.

When the .cmd file is executed, an output folder [26] is created in the project folder ofthe UML-model project; that includes files in which the analysis of the generated test-suites result is shown in the report.html file. In the report.html, there is a list of mutationoperators, most of them are operator replacement, e.g and expressions replaced with others.Total mutants stands for the number of places these mutation operators are possible. Amutant is killed when Momut finds this fault. See figure 31 below, to view the result oftotal mutants killed in the test generation process in Momut::UML.

33

Figure 29: Total Mutants in test-case generation

As seen in the figure above, the total Mutants are divided into categories: Strong Kill(Output), Strong Kill (Failure), Weak Kill, Not discovered and Equivalent. Furthermore,other data used in the tool is presented in the report [26]. The data includes: strategythat is used for test execution, time for the test execution, Momut version, total mutantsand other general information.

The test suites are generated by MoMut::UML when applying the MoMut command-line option -strategy BFS(Breadth First Search) on UML model of WLM system, withtotal 134 Mutants and 47 different Mutation operators. Out of 134 Mutants, 97,10% arefound. In addition, 50% of the found mutants are killed. In figure 30 below, the overviewof test-suites generated by MoMut::UML is presented. Each test-suite implements severalmutants and each mutant is a minor test in its form.

Figure 30: Testsuites generated by MoMut::UML

The Mutation Count for each test-suite illustrate the number of mutants killed in everytest-suite. The mutants are killed by the three different back-ends used in MoMut::UML;Enumerative TCG, Symbolic TCG and Z3, which are described in section 5.3.1. The same

34

mutation operators can be implemented on different transitions and states, thus, each ofthem are counted as a mutant. Each test-suite consists of mutants which are illustrated asseen in figure 31 below, the whole report.html can be found in the output folder[26].

Figure 31: Test-suite detail by MoMut::UML

The position shows where the mutant is allocated in the UML model. Mutation operatoris the applied change, Verdict stands for the status of the mutant, if it is killed or not. Thecontext shows where in UML model the mutant is placed and description is more simplerepresentation of context.

35

6 Results & Analysis

In this chapter, the results are analyzed and discussed in the terms of the performance ofmodel-based testing and the issues that were faced.

6.1 Analysis of Model-based testing

Model-based testing is performed in this work to generate a suite of test cases from theSUT’s behavior. Although it is possible to generate test-cases from this technique, severalchallenges are faced that are presented in this section.

6.1.1 Challenges with model-based testing tool: MoMut:UML

The MBT tool MoMut:UML was chosen by conducting a literature review as illustratedin section 5.2. This tool was most relevant to this work in comparison to other toolsmentioned. MoMut is accessible through a zip file that can be downloaded from MoMut’shome page without any barriers, which makes this tool easy to access for anyone. More-over, a manual is available in the zip file [26] which gives a brief background on MoMut’sfuncionality with a list of command-line options that can be used in the tool, see AppendixB for the options. Although it is an easy to access tool, several challenges are experiencedwhile performing model-based testing. The challenges faced with MoMut::UML are pre-sented in the table below. In which, complication of each challenge is rated 1-5, 1 is leastcomplicated and took less time while 5 is most complicated and took most time.

Table 2: Challenges with MoMut::UML

ID Challenge Complication1 Understand functionality of tool 42 Run the Tool 33 Configuration of tool 44 Connect papyrus to the tool 55 Create UML model in papyrus 46 OCL model language 37 Communication with MoMut-tool creators 58 Test-case generation with the tool 59 Understand the generated mutation data 5

As mentioned, a manual is present for the tool in which all the useful commands and abrief description about the tool is illustrated. However , the manual did not have enoughinformation in order to understand the execution process of the tool and what kind ofoutput the tool generates. To solve the confusion , the demo projects presented in theMoMut zip folder are used. As mentioned in section 5.5, the demo project Alarmsystemis used in order to understand the tools process. Exploratory strategy is used, in whichwe tested out configuration and experimented each step, solved the errors and asked thedevelopers of Momut, until we could proceed through the whole tool execution, which is

36

not easy, and is time-consuming. Anyhow, when the whole process is accomplished, thefunctionality, configuration, and execution of MoMut::UML is understandable, and samesteps are performed in order to generate test-suites for WLM system.

6.1.2 Analysis of UML model

The first step to perform MBT required to build the UML model on an UML modelingenvironment and which could work with MoMut::UML, in our case, Papyrus. As the UMLmodel of the demo project, Alarm System, was already built, further investigation andstudy is required to build the UML model in Papyrus.

The UML model in Papyrus has two parts, a state-machine and a class diagram. Theclass-diagram is necessary to implement as it describes a state machines context and de-fines the static structure. After creating both of the models, it is necessary to connectclass diagram with state machine, which is quite tricky and required literature study andinvestigation, mostly the user manual for Papyrus is used [21]. The configuration of theAlarmsystem UML model is investigated to understand which elements of UML model arenecessary. However, as WLM system is less complex then the Alarmsystem, less elementsare required for the WLM UML model.

The state machine of WLM system is divided into two regions so that they can progressindividually, as the sensor reads the level-measurement continuously and the state tran-sitions are determined continuously based on the read value. The initial state is set totransition to medium state directly, because if the transition is set to e.g. LOW, and theread value from sensor is higher then 40, the system has to transition twice to get into theright state.

6.1.3 Analysis of MBMT Test-suite generation

The output of the test generation with MoMut::UML comes with several files and documen-tation, however, the most important for us is the test-case presentation and test generationanalysis, which is present in file report.html. The test-result is present in section 5.6.A total of 134 mutants are applied to the UML model, which can be interpreted as 134faults on different areas on the UML model that are tested. Which means that the behaviorof the system is tested with 134 different cases. Out of the total mutants, mostly are catego-rized as Strong kill, which means that the behavior(output) of the mutated model differedfrom the original model. Only minor mutants are not discovered. However, 24 mutantsout of 134 are equivalent, which means that they do not really effect the systems behaviour.

Only 3 test-suites are executed by MoMut::UML, however, as seen in figure 31, eachtest-suite beholds several mutants. A test-suite is a sequence of stimulation of expectedobservables. According to MoMut specification, a test-suite is counted as a good test-suitewhen it kills a lot of mutation. According to the creator[23] of MoMut, the test generatorof MoMut has an optimization to choose as few test cases as possible and to cover as many

37

mutants as possible.

In test-suite, the details of each case when a mutant is found is illustrated in figure 31such as; mutation operators and the context where the mutation operator is placed arealso defined.

6.1.4 Drawbacks of Model-based testing

Model-based testing is a testing technique for the future. There is lack of documentationas well as tools to perform this modern technique. The attained result in this work issupposed to ease the concept of MBT by using an available tool. In spite of the fact thatthe tool is not 100 % user-friendly, there is not a visual user interface to run the tool.Additionally the MoMut tool is not fully documented. To finish this project, an assistancewas necessary from the creators of the tool to get better understanding of how MoMutworks and how the generated test cases are analyzed. It took a lot of time to keep thecommunication with MoMut creators. It is not enough to study only MBT, to be good atMBT; a better understanding is needed to complete test case generation process by MBT.For instance, how to create an UML model in a detailed sketch. Even when the wholeprocess of test-case generation is settled, a lot of time is consumed on understanding theoutput of MoMut::UML

6.2 Analyse of the chosen command-line option for MoMut

The chosen command-line option is the strategy BFS(Breadth First Search). The creator’sof MoMut::UML recommend to use this option for the WLM system. The reason behindchoosing this strategy is to cover all the transitions between the different states of the UMLmodel. In this way, the mutations are applied to almost the whole model. In Alarm-systemdemo, another command option is used, step 300. This command is used to search the fulllength of the model, thus the actual performed steps might be higher than this value.

38

6.3 Benefits of MoMut::UML for Model-based testing

MoMut::UML is chosen by delimiting the relevant MBT tools presented in the MBT toollist that is used in this work. Thus, it implements a special technique when applying MBT;it is a model-based mutation testing, which makes this testing tool unique in comparisonto the others mentioned in this work. The test-suites, generated by MoMut::UML doesnot test if the function of the SUT works as they are supposed to, instead it implementsfaults in the SUT, and check if the system reacts to the fault or not. By this, unthought-of errors can be recognized in the SUT. Eventhough no real test-cases are generated byMoMut::UML, it still provides test data with test-suites that verifies the behaviour of theSUT and therefore, is a eligible tool for testing a software system.

7 Threats to Validity

This work required a lot of literature search, as model-based testing is not a clear concept.The search was done in online databases such as IEEE Xplore, Science Direct, and GoogleScholar. All the references, mentioned in this work are from trustworthy websites. One ofthe threats to validity of this work is due to the used MBT tool; as it is not thoroughlydocumented and the result in this work are based on experiments and observation, alsosome guidance from MoMut developer. Also, the amount of MBT tools presented in thiswork is limited, and not all presented tools are analysed, so there is a chance that MBTcan be performed better by a MBT tool that is not discovered in this work.

39

8 Discussion & Conclusion

8.1 Awnsering the research questions

The main purpose of this work is to create a proof-of-concept of model-based testing, MBT,on a water level measurement’s embedded system. Researchers mentioned in the Relatedwork, section 3, have identified MBT as a useful and advance testing approach. Accordingto their work, MBT is a method with a lot of benefits such as reducing the cost of test de-sign, saving time etc. Moreover, using UML with MBT is considered as a valuable idea asit supplies a broad collection of diagrammatic notations and more accurate models can beachieved. With the guide illustrated in this work, a base knowledge on how to implementMBT with MBT tool MoMut::UML is provided to the IT industry. The research questionspresented in section 1.5.2 are possible to answer with the gained result in section 5 andthe analysis in section 6.

• RQ1: What are the steps that should be followed to perform Model-basedtesting with a UML-model?All the steps are presented in section 5. To summarize, the UML model is buildin an UML modeling environment, in our case Eclipse-Papyrus. After the UMLmodel is verified by the modeling environment, a connection between UML modelingenvironment and the MBT tool is established. In our work, the connection is madeby tcg profile which connects the UML model with the MoMut::UML tool, the toolengine is presented as Momut.jar file.

• RQ2: How to use the chosen Model-based testing tool to generate test-cases by using an UML-model of an embedded system?The MBT tool is presented as a jar file. The tool can be executed with a .cmd filein which the command-line option and the path to both UMl model and the jar fileare specified, as seen in figure 17.

• RQ3: How to build an UML-model of the embedded system of water-levelmeasurement system?The analysis on how the UML model is build, presented in section 5.4. In theory, theUML model of WLM system is a simple UML state machine as it only consist of 3states, and transitions in-between as seen in figure 7. However, the UML model builtin papyrus is not only an UML state machine but also a class-diagrams that specifiesthe behavior of the UML state-machine. Moreover, it contains of UML notations,such as signals, events, etc. as presented in section 5.4.

• RQ4: Which type of Model-based testing technique is used by the chosenModel-based testing tool?

MoMut::UML uses model-based mutation testing which functions same as fault-basedtesting criteria, which means that the tool injects faults in the UML-model of WLMsystem. In this way, a fault-based model is created and the tool generates test suitesbased on the differences between the original model and the fault-based model.

40

8.2 Future work

The developers of MoMut::UML are working on new updates on the tool, to mentionfew, they are removing the random strategy and only beholding RRT and BFS. Moreover,they are improving the functionality of the tool by research work on support for symbolicapproaches and researching different sets of mutation operations for different input lan-guages. Also, they are working on a better documentation of the tool. More specifically, aWebAPI for MoMuT will be commercially available in the begining of 2020 which allowsrunning MoMuT on a server and using it from several workstations. The developers arealso working on test case generation for performance testing and for checking safety prop-erties during test case generation. Both are planned for MoMuT 5.0 in spring 2020.This research can be extended in many ways; e.g. by exploring more MBT tools and dif-ferent type of MBT test criterias. Also this guide can be used to implement MBT withMoMut 5.0.

8.3 Conclusion

Model-based testing is a eligible automatic testing technique that can be useful to testsoftware systems and by that minimize testing-cost and time. In this work, only model-based mutation testing is analysed. Momut::UML is a working model-based mutationtesting tool that is easily accessible. It helps to verify the SUT which is the UML modelof WLM system, and discovers errors in the SUT. However, with limited documentation, alot of time was spended on understading the functionality of this tool. Also, the test-suitespresented as the output of the test execution are illustrated unclear, as no literature isfound that describes how the test-suites are analysed. To conclude, the concept of model-based testing is proved in this work and more drawbacks were experienced than benefits.However, a clear guidance on how to implement MBT with MoMut::UML tool is illustratedin this work and the tool does generate test-suites.

41

References

[1] Bruno L., Mark U. "Practical Model-Based Testing: A Tools Approach", MorganKaufmann Publishers Inc. San Francisco, CA, USA. 2007.

[2] Rijwan K., Mohd A., A.K.Srivastava "Generation of automatic test cases with mu-tation analysis and hybrid genetic algorithm", 3rd IEEE International Conference on"Computational Intelligence and Communication Technology" (IEEE-CICT 2017).

[3] Mark U., Alexander P., Bruno L. "A TAXONOMY OF MODEL-BASED TEST-ING", Department of Computer Science, The University of Waikato, Private Bag3105, Hamilton, New Zealand.2006.

[4] Amandeep S. D. "MODEL-BASED TESTING FOR EARLY FAULT DETEC-TION", International Journal of Latest Research in Science and Technology. 2012.

[5] Zoubida A., Yurdaer D. "Development and design of a prototype for monitoring thewater level in water wells using LoRaWAN", Faculty of Technology and Society,Malmö University (2018). 2018.

[6] Chen M., Qiu X., Li X. "Automatic Test Case Generation for UML Activity Dia-grams", Postprint Version. International Workshop on Automation of Software Test,Pages 2-8. 2006.

[7] D.M. Rafi,K.R.. Moses,K. Petersen, Mika.V. Mäntylä "Benefits and limitations ofautomated software testing: systematic literature review and practitioner survey",Blekinge Institute of Technology, Karlskrona, Sweden, 2012.

[8] B. Meyer, A. Fiva, I. Ciupa, A. Leitner, Y. Wei, E. Stapf "PrograMs ThaT TesTTheMselves", Computer, 2009 - ieeexplore.ieee.org.2009

[9] M. Prasanna, S.N. Sivanandam, R. Venkatesan R. Sundarrajan "A SURVEY ONAUTOMATIC TEST CASE GENERATION", Department of Computer Science andEngineering PSG College Of Technology, Coimbatore, 2005.

[10] Ina S. "Model-Based Testing", IEEE Software (Volume: 29 ,Issue: 1 , Jan.-Feb. 2012),IEEE Computer Society , 2012.

[11] Mohammad M. E. "THE SURVEY OF MODEL BASED TESTING AND INDUS-TRIAL TOOLS", Department of Computer and Information Science, LinköpingUniversity. 2008.

[12] F. Bouquet, C. Grandpierre, B. Legeard, F. Peureux, N. Vacelet, M. Utting "A subsetof precise UML for model-based testing", Proceedings of the 3rd international work-shop on Advances in model-based testing Pages 95-104 London, United Kingdom.2007 .

42

[13] Tommi T., Mika K., Julian H. "Experiences of System-Level Model-Based GUI Test-ing of an Android Application", 2011 Fourth IEEE International Conference on Soft-ware Testing, Verification and Validation, 2011.

[14] Bill H., Helmut G., Klaus B. "Model Based Testing of System Requirements usingUML Use Case Models", 2008 1st International Conference on Software Testing,Verification, and Validation. 2008.

[15] S. Weißleder, H. Schlingloff "An Evaluation of Model-Based Testing in EmbeddedApplications", 2014 IEEE Seventh International Conference on Software Testing,Verification and Validation. 2014.

[16] Burstein, F. "System development in information systems research". Research Meth-ods for Students, Academics and Professionals, 2nd ed., National Library of Aus-tralia: Quick Print, Wagga Wagga, 2002, pp. 147-158. Available: Science Direct,https://www.sciencedirect.com/science/article/pii/B9781876938420500083.

[17] Nunamaker, J. F., & Chen, M. "Systems Development in Information SystemsResearch". Proceedings of the Twenty-Third Annual Hawaii International Confer-ence on System Sciences, vol. 3, pp. 631-640, 1990. [Online]. Available: IEEEXplore, http://ieeexplore.ieee.org/stamp/stamp.jsp?arnumber=205401&tag=1. [Ac-cessed February 22, 2019].

[18] Micskei .Z, Model-based testing (MBT). Available: http://mit.bme.hu/~micskeiz/pages/modelbased_testing.html#tools. [Access: May 1, 2019].

[19] MBTsuite, MBTsuite - The model based testing tool. Available: https://mbtsuite.com/. [Access: May 1, 2019].

[20] PolarSys, PolarSys, Open source solutions for embedded system . Available: https://www.polarsys.org/. [Access: May 1, 2019].

[21] Eclipse, Eclipse documentation - Current Release. Available: https://help.eclipse.org/neon/index.jsp?topic=%2Forg.eclipse.papyrus.uml.doc%2Ftarget%2Fgenerated-eclipse-help%2FPapyrusStarterGuide.html. [Access: May 3, 2019].

[22] Bernhard A., Harald B., Elisabeth J., Willibald K., Rupert S. & Stefan T. "Mo-MuT::UML Model-based Mutation Testing for UML", AIT Austrian Institute ofTechnology. 2013.

[23] Meeting with Momut-tool creators

[24] Andreas F., Willibald K., Rupert S., Thorsten T., George W. Model-based, mutation-driven test case generation via heuristic-guided branching search, MEMOCODE 2017,Vienna, Austria

[25] MoMut. Available: https://momut.org. [Access: May 3, 2019].

43

[26] MoMut::UML,Output of MoMut::UML https://drive.google.com/drive/folders/1_4hmVQbnXZmH6tHYCD9aoM7nki7oI97u?usp=sharing

44

9 Appendix A

Figure 32: Activity diagram for the water-level measurement system [5]

45

10 Appendix B

Figure 33

46

Figure 34

47

Figure 35

i

Figure 36: Operations List for running Momut.jar

ii