rational israel users conference automated model driven software testing rational users conference...

Rational Israel Users Conference

Automated Model Driven Automated Model Driven Software TestingSoftware Testing

Rational Users Conference

Tel Aviv

June 7 2004

Alan Hartman

IBM Israel – Haifa Research Laboratory


AgendaAgenda

• AGEDIS Project Overview• Architecture• Methodology• Interfaces• Tools• Future Plans


AGEDIS OverviewAGEDIS Overview

• Automated model-based test Generation and Execution for DIStributed systems

• Methodology and tools for model-based testing• Open interfaces• Mixture of academic and industrial partners• Three phase timetable of experiment and

development• November 2001- February 2004• Partially funded by the European Commission


Consortium PartnersConsortium Partners

• IBM Haifa Research Lab• Oxford University• VERIMAG/IRISA• Imbus• France Telecom• IBM UK• Intrasoft International


Methodology & Architecture Methodology & Architecture

• UML based component testing

• Compliant with RUP• High degree of

automation• No silver bullet


Automated Testing MethodologyAutomated Testing Methodology


Automated Testing ArchitectureAutomated Testing ArchitectureG

UI

& P

rodu

ctiv

ity

Aid

s Model

GenerationDirectives

ExecutionDirectives

Compiler IntermediateFormat

Simulator

Generator

AbstractTest Suite

Suite Execution

Trace

Analyzers

VisualizerEditor

Execution


BenefitsBenefits• Starting from specification

– Involves testers early in the development process – Teams testers with developers– Forces testability into product design

• Building behavioural model and test interface– Finds design and specification bugs - before code exists– The model is the test plan - and is easily maintained

• Automated test suite generation– Coverage is guaranteed - increases testing thoroughness– Matches coverage goals to testing budget– Zero test suite maintenance costs

• Automated test suite execution– Finds code and interface bugs– Includes a framework for the testing of distributed applications– Reduces test execution costs


InterfacesInterfaces

• UML Profile for AGEDIS• Test Generation Directives• Test Execution Directives• IF Model Execution Interface• Abstract Test Suite• Suite Execution Trace

UML

TGD

TED

IF

ATS

SET


User Modeling InterfaceUser Modeling Interface

• The AGEDIS Modeling Language is a profile for UML 1.4:– UML Class diagrams - structure– UML Object diagrams - snapshots– UML State diagrams – behaviour & test

purposes

• Annotated with an action language – IF


Test Generation DirectivesTest Generation Directives

• Test Purposes – stereotyped system level state diagrams

• Coverage Directives – stereotyped system level state diagrams

• Coverage without tears – 4 levels – parameters to the test generation algorithm


Test Execution DirectivesTest Execution Directives

• XML Schema– Model translation from model abstractions to

SUT concrete implementations– Test architecture

• Host addresses

• Delays, timeouts

• Prologues, epilogues


IF - Model Execution InterfaceIF - Model Execution Interface

• IF Encoding of the model– Classes– Objects– State machines


Test Suite and Trace InterfaceTest Suite and Trace Interface

• XML schema – for test execution and tracing

• Model description– classes : constants, types, control & observable

signatures– a special class is defined for the tester– object identities

• Test Suite - set of test cases• Test Trace – record of executed test cases


Test Case and Test TraceTest Case and Test Trace

• Interaction graphs - between the tester and the SUT

• Associated verdicts (Pass, Fail, Inconclusive)

• Alternate paths for non-determinism in test cases

• Expected results in test suite• Observed results and verdicts in trace


Abstract Test SuiteAbstract Test Suite


ToolsTools

• User Interface• Modeler & Model Compiler• Model Simulator• Test Generator• Test Execution Engine• Test Suite/Trace Viewer/Editor• Feedback & Analysis• Bug Reporter• Report Generator

UI &

RG

CSG

FA

VEEE


GUIGUI


ModelingModeling Tool & CompilerTool & Compiler

• Objecteering UML modeling tool

• Tool profile to convert to XML

• General purpose XML to IF compiler– Written in Java, with

XMI in mind as a future input format


Model SimulatorModel Simulator


Test GeneratorTest Generator

• Based on GOTCHA and TGV

• GOTCHA – uses Murphi specification language

– explicit traversal of state space

– extensive coverage criteria

• TGV– language independent simulator

– focus on distributed applications

– explicit test purposes as sequences of interactions


Test Execution EngineTest Execution Engine– Input: Abstract Test Suite and

Test Execution Directives– Output: Suite Execution Trace– Multiple platform (e.g. Linux,

Windows) support– Multiple external interface

(Java,C++,C) support– Test distribution + centralized

logging– Interactive test case execution– GUI to view execution progress,

inputs, and outputs


Execution Engine ArchitectureExecution Engine Architecture

N e t w o r k

Test SuiteDriver

I/O

HostManager

ProcessManager

1 1 on each host 1..N on each host

. . .

Object

Object


Feedback ToolsFeedback Tools• Coverage analysis

– Detect uncovered areas of the model in either test suite or test trace

– Create test purposes to reach them– Invokes a functional coverage tool from

www.alphaworks.com

• Defect analysis– Clustering of defects– Feature extraction from clusters– Create test purposes to reproduce the bug


AGEDIS Future PlansAGEDIS Future Plans


Future PlansFuture Plans

• Exploitation Activity– Available under license for non-commercial use– Available as a service

• Incorporation in wider Model Driven SE Effort– Integrate methodology with RUP– Migrate to Eclipse, XDE– Focus on business integration domain


ModelWareModelWare• New European Project• Model driven software engineering• Eclipse based tools for UML modeling, and the

full MDA lifecycle– Model driven architecture

– Platform independent and platform specific models

– Model simulation

– Model transformation


Thanks to:Thanks to:• A. Ramfos., S. Liapis, N.

Giannelos, A. Hondouridakis, M. Sardis, K. Bechrakis, V. Akousi-Krivki

• K. Dussa-Zieger, J. Trost, B. Nossem, T. Linz, B. Mattern, J. Hofer, H. Raessler, T. Rossner,

• I. Craggs, I. Griffiths• Y.-M. Quemener, D. Vincent,

T. Heuillard, N. Moteau• A.Bertolino, A. Wills, S. Reid

• K. Nagin, O. Edelstein, A. Kirshin, S. Olvovsky, M. Berg, L. Raskin, T. Shiran, C. Sacharen, M. Barshay, D. Neimer

• L. Mounier, M. Bozga, Y. Lakhnech

• T. Jeron, E. Demairy, V. Tschaen

• J. Davies, A. Cavarra, C. Crichton, J. Woodcock, M. Field

rational israel users conference automated model driven software testing rational users conference...

Documents