From Keywords toModel-Based Test

AutomationTuula Pääkkönen, Nokia

& Mika Katara, Tampere Univ. of Technology,

Finland

T2

2

From Keywords to Model-Based Test Automation

Tuula Pääkkönen

Global Concept Owner, Test Automation

Nokia Technology Platforms

Finland

Mika Katara

Dr. Tech., Adjunct Professor (Docent)

Tampere University of Technology, Institute of Software Systems

Finland

3

Outlook

1. Introduction to the anticipated benefits of model-based

software testing

2. Combining keywords and model-based testing in the

Symbian S60 GUI testing domain

3. Introduction to the associated open-source tool set and

experiences from industrial case studies

4. Conclusions

4

Model-Based Testing (MBT)

• MBT automates not only the execution of tests but also

their generation

• It is not a new idea to use formal models, for example finite

state machines, to generate black box tests

• The topic has been widely studied in the academia over the

last decades

• Industrial deployment has been successful in certain

domains (see, for instance [1,2])

5

MBT…

• However, large scale deployment of MBT has turned out to

be very difficult (see, for instance [3])

• Tools can be hard to use

• Managerial problems

• Re-organization of work etc.

• Adjusting the testing processes (agile etc.)

• Modeling is not easy, not everybody can do it

• Model modularity and management

• (100 states vs. 100^5 states)

6

Benefits of MBT

• Conventional test automation is not particularly good in

finding new bugs

• Mainly used for regression testing to make sure nothing

already tested has been broken while adding new features /

fixing bugs / refactoring

• MBT can generate new tests in every test run

• Even a single loop in the model (state machine) makes it

possible to generate an infinite number of different tests

(based on the number of loop iterations)

• Much higher coverage can be achieved than with

conventional techniques

• Note: structural coverage based on model elements does not

necessarily correlate with the code coverage

7

Benefits…

• Maintenance of conventional test scripts, especially in GUI

testing, can be very time consuming

• Since in MBT test are generated automatically, it is

sufficient to update the models and generate the tests

again

• The larger the test suite, more anticipated savings

compared to the maintenance of scripted tests

8

Capture Replay,

Spaghetti Scripts

Structured Test

Scripts

Data-Driven

Scripts

Keywords,

Action words

Generations of (System Level) Test Automation

See for instance, Software Test Automation:

Effective use of test execution tools

By Mark Fewster and Dorothy Graham,

Addison Wesley, 1999.

9

Manual

testing

Scripted

test automation

Model-based

testing

Generations…

10

From Keywords to Model-Based Automation

• A state machine is a collection of states and transitions

• Transitions describe how the state changes occur

• One of the states is marked as the initial state

• In his article [4], Hans Buwalda suggested using spread

sheets for combining state machines with action/keywords

• However, we prefer to present the models more visually

and modularizing large models using parallel composition

of smaller component models

11

On-line vs. Off-line Testing

On-line?

Generate

Test Suite

Execute

Test Suite

Evaluate

Test Results

No

Select Next

Test Step

Execute Step on

Model & SUT

Evaluate

Result

Objectives

Achieved?

Report

Results

Yes

No

Test

Behavior

Test

Objectives

Yes

Alan Hartman, Mika Katara, and Sergey Olvovsky: “Choosing a test

modeling language: a survey”. Proc. Haifa Verification Conference

2006, IBM Haifa Labs, Haifa, Israel, October 2006. Number 4383 in

LNCS, pp. 204-218. Springer 2007.

12

Challenges in Industrial Adaptation

• Requires commitment to MBT, at least in testing / test

automation organization

• See Abrahamsson [5] for commitment development in

process improvement activities

• Increased need for communication at lower levels, when

test models are built to be as comprehensive as possible

• The most probable scenario is that the testing organization

will have new roles and tasks incorporated in the traditional

software testing activities

• Model creation requires effort

• However, [6] reports that MBT-techniques can be learned

within a few days

13

Challenges…

• The main challenge is how to take the step from pilots to

deployment:

• This has a vicious circle: someone should be the first to

show-case the results, but on the other hand, the early

innovators should get the benefits relatively fast, which may

turn out to be difficult to achieve

• In addition, scalability of the chosen MBT solution to real-life

situations needs to be ensured [7]

• The role of design models, if such exist, is still unclear in

MBT

• Adopting MBT could trigger a change in the whole software

development towards model driven approach

14

Domain-Specific Approach – S60 GUI Testing

• Our domain: GUI testing of Symbian S60 smart phones

• Our solution is based on a domain-specific test modeling language [8] instead of UML or any other generic language

• This enables higher level of automation and usability

• “MBT-thinking” helps in partitioning the target of testing into manageable component models

• Testing complex interactions between different applications and systems

• Managing complexities within functional, regression testing goals in a product family context

• Performance aspects – monitoring longer sequences –being able to stress the system under test for longer periods of time

15

Domain-Specific…

• Action words describe user’s actions at a high level of abstraction

• Send an SMS, answer a call, add a new contact, browse the calendar etc.

• An action word is translated to a sequence of keywords(key strokes) for menu navigation, text inputting etc.

• Action machines containing action words are composed with refinement machines containing keywords

• The resulting composite model is input to the tools executing the model, i.e. generating sequences of keywords

• To avoid state space explosion, the test generation has been implemented using an on-the-fly algorithm

16

Additional Benefits

• Testing the interactions between the different smart phone

applications is seen as additional benefit compared to

existing methods

• Test data universe – being able to include more versatile

test data in the test generation/execution phase

• Decisive test ending criteria based on test coverage

• With the help of the test model the coverage can be

visualized and utilized in the test generation/execution phase

17

Example Test Models

S60 Camera application, action word model

Illustration: Antti Kervinen/TUT

18

Example…

S60 Camera application, keyword model

Illustration: Antti Kervinen/TUT

19

TEMA tool set

Design Tool

Adapted Test Tool

Commercial test

execution tool

Get keyword

Model Execution Tool

kw

Test Control

Test Engine

select

GuidanceCoverage

Getguidance

Covered?

Test Control Model Test Model

Get control

Get data

Model Designer

create

RefinementMachine

Model Composer

create

Test Data

create

Get test

ToolAdaptation

Test Modeler

execute

Action tier

Test Log

statusSUT 1

SUT N

TestVisualizer

Executioninfo

select

instantiate

Recorder

Test steps

Control tierTest Manager

Test Model ExecutionSpecialist

KeywordLibrary

status

kw

observe

ActionMachine

Illustration: Mika Maunumaa/TUT

20

Experiences from Industrial Case Studies

• A lot of problems with building tool adaptation

• Certain keyword implementations have turned out to be very

problematic

• Due to this, test execution has been delayed

• In the first case study, the SUT was a S60 phone already

released to the market and the study concentrated on the

built-in applications

• Some defects were found during test modeling

• The most severe of these caused one application to crash

21

Experiences…

• Due to the aforementioned technical difficulties, limited test

execution become possible only in the second case study

• This time the SUT was a third-party S60 application

• One defect was found in actual test execution and few other

issues while developing the test models

• None of these defects can be considered serious

• In both studies, the SUT has been mature and very well

tested before the case studies

• Most of the defects found have been concurrency related,

which supports our earlier findings [9]

22

Experiences…

• Creating the test models for the basic S60 applications has

been easier than anticipated

• Dozens of component models have been created for S60

built-in applications, such as Camera, Calendar, Contacts,

Messaging etc.

• Even domain-specific models can be hard to understand

• In the third case study, a web interface is to be used for

specifying test purposes (coverage requirements) that

drive the test generation [10]

• This allows hiding the complexity of the solution (test models

and generation algorithms) from the testers

23

24

Conclusions

• Deployment of MBT might be hard… but the benefits are

really promising

• Upgrading from keyword automation to MBT automation is

possible

• Building a suitable tool adapter may turn out to be very

difficult

• Test modeling reveals errors effectively

• A domain-specific approach needs custom tools, but

promises higher level of automation and usability than

generic approaches (UML etc.)

• TEMA tools are available under the MIT Open Source

license, see http://practise.cs.tut.fi

25

References

[1] Campbell C, Grieskamp W, Nachmanson L, Schulte W, Tillmann N,

Veanes M: Testing concurrent object-oriented systems with Spec

Explorer. Proc. Formal Methods 2005. no. 3582 in LNCS. Springer.

[2] Utting M, Legeard B: Practical model-based testing – a tools approach.

Morgan Kaufmann, 2007.

[3] Robinson H: Obstacles and opportunities for model-based testing in an

industrial software environment. Proc. 1st European Conference on

Model-Driven Software Engineering (2003). Nuremberg, Germany.

[4] Buwalda H: Action figures. STQE Magazine, March/April 2003.

[5] Abrahamsson P: Commitment development in software process

improvement: critical misconceptions. Proc. 23rd International

Conference on Software Engineering (ICSE 2001), Washington, DC,

USA, 2001. IEEE CS.

[6] Farchi E, Hartman A, Pinter SS: Using a model-based test generator to

test for standard conformance. IBM Systems Journal, 41(1):89–110,

2002.

26

References…

[7] Lakey PB: Model-based specification and testing applied to the ground-

based midcourse defense (gmd) system: an industry report. Proc. First

International Workshop on Advances in Model-Based Testing (A-MOST

2005). ACM Press.

[8] Katara M, Kervinen A, Maunumaa M, Pääkkönen T, Satama M: Towards

deploying model-based testing with a domain-specific modeling

approach. Proc. TAIC PART - Testing: Academic & Industrial

Conference, Windsor, UK, 2006. IEEE CS.

[9] Kervinen A, Maunumaa M, Pääkkönen T, Katara M: Model-based testing

through a GUI. Proc. 5th International Workshop on Formal Approaches

to Testing of Software (FATES 2005). No. 3997 in LNCS. Springer.

[10] Katara M, Kervinen A, Maunumaa M, Pääkkönen T, Jääskeläinen A:

Can I have some model-based GUI tests please? – Providing a model-

based testing service through a web interface. Proc. second annual

Conference of the Association for Software Testing (CAST 2007),

Bellevue, WA, USA, 2007.

27

Thank you for your attention

Contact:

Tuula Pääkkönen Mika Katara

Nokia Technology Platforms Tampere Univ. of Tech.

tuula.a.paakkonen@nokia.com mika.katara@tut.fi

Special thanks to TEMA team at TUT: Antti Kervinen, Mika Maunumaa,

Antti Jääskeläinen, Merja Leppänen, Henri Heiskanen, Jukka Mikkonen