automatic test case generation
DESCRIPTION
Automatic Test Case Generation for Function Block DiagramsTRANSCRIPT
Automatic Test Case Generation for Train Control Systems Adnan Causevic
TOCSYC Industrial Workshop, 2014-11-13, Västerås
2
Today’s talk is about…
● Software Testing Laboratory @ MDH
● Research: ● Test Automation ● Quality of Tests ● Developers Testing
● AGENTS - KKS funded project ● Bombardier Transportation ● Maximatecc
● Involved in the TOCSYC research environment
● Intelligent Embedded Systems Master Program Coordinator
3
About me
● ATAC research project ● 2011-2014 ● Funded by ITEA2/Vinnova ● Advanced Test Automation for Complex Software-
Intensive System
4
Cooperation with Bombardier
“The project has brought test automation on the agenda of European industry.”
- ITEA3 Final Review Report
● Train Control Management System (TCMS) ● a high capacity, infrastructure backbone ● the center of the distributed system
● Functions controlled by TCMS include ● collecting line voltage, ● controlling the train engines, ● opening and closing the train doors, ● upload of diagnostic data.
5
About Bombardier
● Vehicle ● Vehicle Integration testing
● System ● System Integration testing
● Software ● Modules integration testing ● Modules testing
● “Get to the driver’s perspective as soon as possible”
6
About Bombardier … and their testing
Responsibility of a Developer!
● IEC 61131-3 ● Function Block Diagram ● Graphical programming language ● Based on element composition ● Various predefined elements
7
Development Environment
● Coverage ● Strict measurements mandated by safety standards
● Tooling ● No tool support to measure coverage on FBD
● Time ● Creating tests just to reach the coverage leaves very little
time for functional testing
8
Current challenges … with module testing
9
CompleteTest
10
Research approach FBD Program
Transformation 1 Annotation2UPPAAL timed
automata
...
Reachability properties
∃ ♢ β ,∃ ♢ pi ,
∃ ♢ c0 and c1
Test traces Test cases(Step) (1)(Time) (20s)(Inputs) (0 1 23)(Outputs) (1 2 543,9)
UPPAAL model checker
Test Generation3
Logic-‐‑based coverage criteria
DC, CC, MC/DC
”Automated Test Generation using Model-Checking: An Industrial Evaluation”, Eduard Paul Enoiu, Adnan Causevic, Thomas J. Ostrand, Elaine J. Weyuker, Daniel Sundmark, Paul Pettersson. International Journal on Software Tools for Technology Transfer, 2014, Springer.
● Time required to generate tests ● satisfying the DC, CC and MC/DC logic coverage criteria
11
Case Study at BT
● For 34 of the 157 programs, the tool did not terminate after running for a substantial period of time.
● Cut-off time was set to 10 minutes.
● There is an indication that as the number of decisions increases, the performance deteriorates and the cost of using the tool may become prohibitive.
● But, if you have so many decisions, how would you test it manually?
12
Case Study at BT (2)
● Should a developer generate the tests from scratch? ● Should a developer provide manual tests first? ● How many (or how much in terms of coverage)?
● When should a developer use the Validate Test button?
● Should developers use this tool at all or should we automate that as well (nightly build, etc.) ?
13
Best Practices?
● Internal behavior of each function block has to be modeled ● Currently, blocks from the standard language are modeled ● and a few custom made by Bombardier which are often used
● State-space explosion ● a known problem of model-checking ● large number of Boolean or Integer input values ● but, again, how would you do it manually ?
14
Limitations
● Complementing model-checking with: ● Static analysis of FBD’s ● Search-based software testing
● Trying some new approaches: ● Different model-checkers, SMT solvers, …
● Measuring effectiveness of generated tests ● In terms of how good they are in finding faults
● Experimenting with FBD programs from
other organizations
15
On-going and Future work
● Adnan Causevic [email protected]
● Eduard Enoiu [email protected]
● www.completetest.org
16
Questions?