©2010 Nokia. 1

Experiences from GUI Test Automation in Agile Development Ville Ilvonen Principal Engineer, Tools Development, MeeGo Computers, Nokia September 23rd 2010 Seminar on Model-Based Testing of Smartphone Applications

©2010 Nokia 2

Ingredients

•  Model-Based Testing project funded by Finnish tax payers and companies (Thanks TEKES et al.) •  Relatively large agile organization in Nokia •  Continuous integration system to run automated tests •  Bunch of developers and testers to pilot project tool set – driven by Maemo 6 and smartphone R&D. •  Experimental GUI test automation tool for Qt •  World-class expertise on MBT from TUT

©2010 Nokia 3

Recipe

Mix all ingredients in good proportions with values from agile manifesto and serve chilled.

Enjoy with your collegues.

©2010 Nokia 4

Quotes from the kitchen ”Ambitious target of our pilot would be to improve MBT awareness. We’d also like to deploy practices and toolset in agile development organization. Maybe

also execute those test models in continuous integration system – let’s say over night using path algorithm of choice.”

Yours Truly

”Here’s flow diagram and architecture of our CI system. We run huge amounts of traditional tests in it. Isn’t it impressive?”

Yours Truly

”To do modeling properly we’d like to have full UI specifications for this unpublished product, please.”

TUT project personnel

Disclaimer: Author didn’t record any discussions so quotes are not perfectly accurate

©2010 Nokia 5

Ideal Catering Proposal

Courtesy of Mika Katara et al.

©2010 Nokia 6

Some Activities in Agile Kitchen (+ lots of other roles not described here) •  Developer

•  Works closely with UI designers, architects and test engineers in scrum team. Writes code and tests based on user-stories. Splits stories to tasks in sprint planning. Sprints end to deliveries to product release content. Follows Definition of Done. The most advanced ones design software using test driven development (TDD).

•  UI Designer •  Creates UI designs and specifications. Provides input for

developers. Provides feedback based on delivered content. •  Test Engineer

•  Tests with developers during sprint or tests delivered users stories from release content. Finds and reports bugs.

•  Test Designer •  Designs test asset libraries to cover testing of delivered features •  Some of the GUI tests end up automated based on planned

utilization • Test Manager

•  Decide which tests to run over given time slot to find bugs and reduce risk

Run development of new features in sprints (2 weeks).

Get feedback from testing and users, try concepts and improve design.

Embrace change. Change is good.

©2010 Nokia 7

Quotes from the kitchen - Continued ”Model-based testing, that sounds interesting but we have these test asset libraries which contain huge amounts of

tests. How do you describe run rates and pass rates with models? We need to keep operations running.” Test Designers and Test Managers

”These test modeling tools work pretty well but they have some rough edges from usability point of view.” Piloting Developers

”Models that map states and transitions to detailed application components for interaction and verification are pretty tough to maintain when application changes, some framework library changes and test tool also changes. That’s the

case with traditional tests as well but models kind of add a layer on top of that.” Piloting Developers

”Formalism in models is an over-kill.” Piloting Test Engineer

”I don’t mind formalism but maintenance of models during sprints is tough.”

Piloting Developer

”Training was useful, very interesting with good hands-on part. Nice to try with products we work with.” Many participants after training provided by TUT researches

Disclaimer: Author didn’t record any discussions so quotes are not perfectly accurate

©2010 Nokia 8

How about MBT fast food instead of French Menu? Examples of tests in asset library

Test plain HTML with browser X 1.  Open browser X 2.  Type URL 3.  Verify plain HTML content 4.  Close browser

Test flash with browser X 1.  Open browser X 2.  Type URL 3.  Verify that flash works 4.  Close browser

Test SVG functionality with browser X 1.  Open browser X 2.  Type URL 3.  Verify that SVG content is

rendered correctly 4.  Close browser

Open Browser

X

Type URL

Verify plain HTML Verify

Flash

Verify SVG

Close browser

Same tests in example model

If I have a model which has – let’s say 150 use cases – how can I report test coverage and where the bugs are?

©2010 Nokia 9

New Menu

•  Customization of ideal catering proposal per organization but keep it interesting from research point of view. •  More user-friendly notation and toolset with manual testing support for GUI testing. •  Realization that majority of GUI testing during product development is not automated but manual. Manual adapts to change very fast. GUI automation is tempting but making it affordable with embraced change is not easy. •  Test managers expect practical model-based-test reporting solutions with executive summaries.

©2010 Nokia 10

Best Tastes

•  Targets? Got some, lost some. •  Toolset provided by project is fairly robust but not a product. •  Possible to convince suits to finance further development? Maybe, doesn’t look hopeless. •  One of piloting developers already wrote MBT-adapter to simpler GUI testing tool (Thanks Riku). •  SW engineering project group at TUT took challenge to create tool for manual test modeling. Let’s see what comes from that. •  Applying MBT to agile RD is completely different from testing of released products. As expected. •  Discussion to continue project already started. Discussed here?

•  I’d love to see this catered with MeeGo.

©2010 Nokia 11

References and other links of interest

•  TEMA toolset - http://www.cs.tut.fi/~teams/download.html •  NOTE - Internal version was used during the pilot

•  TDriver tool - http://gitorious.org/TDriver •  TDriver for Documentation MeeGo - http://wiki.meego.com/Quality/QA-tools/TDriver •  On MBT adapter for QtUITest - http://gitorious.org/qa-tools/qtuitest-mbt-adapter •  On Maemo 6 and MeeGo - http://wiki.meego.com/MeeGo_and_Maemo6 •  TUT SW Engineer Project (OHJ-3500) - http://www.cs.tut.fi/kurssit/OHJ-3500/aiheet10.html

•  Project proposal 6 (only in Finnish) – One group selected the project. Option for MeeGo Award. •  Info on automated testing system targeted for use in pilot

•  http://conference2010.meego.com/session/automated-testing-solution-meego (Proposal)