Testing JavaFX UI
Tips from JavaFX quality team

Alexandre (Shura) IlineJava and JavaFX quality architectOracle

Agenda

UI testing overview

Test automation approaches

Jemmy library

Test automation effectiveness

Test library design approaches

UI testing by Wikipedia

GUI software testing is the process of testing a product that uses a graphical user interface, to ensure it meets its written specifications.

UI testing most often ...

Checking whether usage of a product UI leads to results expected by the the person who performs testing

Start text editor

Push File/Open

Verify file chooser directory

Select some file

Verify editor area content

Verify application title

Verify buttons availabilities

....

Note that in reality all the verifications should be done throwg waiting

UI TestingManualAutomated

Initial stepDesign test specification.Create tests.

Establish regular runs.

OngoingClick, click, clickAnalize results

File bugs

Change the specificationChange the tests

QualificationHigh, low for test executorsHigh

EffectivenessLowHigh

OtherInexpensive to startContinuous quality monitoring , inexpensive to reuse

FunBo-o-o-ringMuch like programming

Effectiveness is estimated under assumtion that the testing is thoruogh and there are many tests

DevelopmentContinuous build

CommitBuildExecuted automagicallyafter commitSuccessAnalysis.Rollback!!!

Code changes

No

Yes.

PromoteCode is compilableTest further

This is a well accepeted practice nowadays.However it is not really possible without testing.And it is not really possible with manual testing as the turnaround is too long to rely on.

Continuous build with testing

CommitBuildSuccessTestingIs it working?PassedAnalysis.Rollback!!!

Code changesTestchanges

No

No

Test furtherBuild is goodCode line is healthyGo on ...

Yes

Yes. = Compilationsuccessful

This is a well accepeted practice nowadays.However it is not really possible without testing.And it is not really possible with manual testing as the turnaround is too long to rely on.

Automation approachRecord && ReplayCoding

Test creationInexpensive*Usually just repeating manual test in special environmentMust be accompanied by other meansExpensive*Consists of programming

Test executionDoes not depend on approach

Test maintananceHigher (in most cases)*Very much depends on test formatLower*Depends on principles of building test library

Test analysisDoes not depend on approach

(*) Much more information closer to the end of this presentation

If we get to it, I will be howing figures which explain why do we think so

Jemmy

Jemmy v2Started as a tool to tests TeamWare UI (1999)Used for NetBeans extensions (2000)Official test tool for NetBeans (2001)Open-source (2001)

Jemmy v3Started in (2008) as a proof of concept experiemntExtended to support JavaFX (2009)Opensource with support of JavaFX 1.2 (2009)Developed in close-source for 1.3 since then

It is not a toy

Jemmy usages

V2 (Swing/AWT)Internal: Swing, NetBeans, Glassfish

External: Used by: Boeing (x2), agfa.com, compuware.com, attix5.com, exgate.tek.com, globo.com, gmx.de, non.agilent.com

Evaluated by: syntelinc.com, covast.com, visus-tt.com, covast.com, nokia.com, twt.com, ubs.com

V3 (FX)Internal: JavaFX, Authoring Tool, JavaStore

ditto

Demo...Finally!

Jemmy v3

JavaFXAWT/SwingSWTScenegraphLCDUIJemmyCoreInterfacesAWT robotFX robotHierarchyInputScene, nodecontainersScroller, slider, list, text, ...GenericinterfacesCovered in this presentation

UI test

FindDoVerify

PassPassPass

FailFailFail

Failure analysis

Find next controlTo perform operationOn inVerify that expectedState reachedPerform necessary actions

Lookup

By IDEasiest but may not be possible

open_file_btn

By typeMost common

Button, text, combo box, etc

By indexUnavoidable

2nd button with text browse

By toString()Sucks

projectmanager.Main$Main$Script$1Scene$ObjLit$20@4d16318b

By text, tooltip, associated labelBest, if possible

Button with text browse

UI could bevery complicated

UI could bevery complicated

UI could bevery complicated

Lookup demo

Interfaces

InterfaceControl typesDescription

Mouse, Keyboard, DragEnythingLow level input

ParentContainers, list (for its content)Something you could look within

SelectableToggle button, radio button, combobox, check box, lists, etcA control which provides limited number of choices

CaretOwnerText box, scroll bar, sliderA control which has a number value which changes within some range

TextText boxEditable text container

Interfaces demo

Verifications

UI feedback

Non UI feedback

Dialog displayed

Test changed

Image updated

Progress bar changed position

File created

Database updated

Sunset happened :)

Waiting

UI is a multy-thread environment

Things happen in backgroundTest code is in another threadNot much could be really verified

Everything should be waited for

Verification demo

Test automation effectiveness

TD

+

*

TS

NR

TM

*

NR

NC

*

EA

=

NC

*

EA automation effectiveness

NR and NC are characteristics for a product.TM is a characteristic of a test suiteTD and TS depend on test automation approach

Smaller TD and TS - higher the EA.

Assumptions: TM = 1 engineer*week TS = 0.1 * TM TD = 5 * TM NR = 8

Td or Ts what to minimize

TS - if (NC * NR) is big Multi-platformCompatibility with external products (servers, browsers, ...)Long-living

TD - if (NC * NR) is small Proof of conceptPreview

Tests fail every now and then ...

because the tested UI is changed

Tests fail every now and then ...

Ah! And also because errors are made ...

End2End demo

Ts continuends

Time spent onWhat to do%% of time

Allocating failuresUse test harness1% - 5%

Anderstandint the failure reasonUse test logging, save images, save UI state10% - 80%

Fix the testsMove common code to the library (*)90% - 10%

(*) The only way it is different from programming is that there are a lot of tests. More on this later.

Harness

TestNG

Understanding failure

LoggingUI operationsGoing to push ... buttonSelecting ... menuUI responcesFound list view: UI hierarchyScene Group Button

UI statesImages

DemoJemmy logging functionality

Fix the tests

One UI, many-many tests.

Do as few test code changes as possible

Ideally one change in test code for one change in UI code.

Decrease maintenance cost

Implement reusable operations in a library

openFile(String filename)Decrease implementation dependencies

lookup(Button.class, new ByText(Low)) Increasing stability

.wrap(0).mouse().click();Thread.sleep(1000);...

DemoRefining test.

Library design approaches

Remember the formula?

TD

+

*

TS

NR

TM

*

NR

NC

*

EA

=

NC

*

EA automation effectivenessTo be used for every particular product.

NR and NC are unique for a product.TM is a characteristic of a test suite.Smaller TD and TS - higher the EA.

Coefficient depend on the way you write your tests

TS mainly consists of time for test modification.When product changes, tests need to be changed accordingly.Many tests! Hundreds.Less changes of test code per a change in the product UI.Ideally ... no more than one.But ... how? You are the coders you know:Move code to test library.

Application domain model

Car recordCar recordColorModelMakeYearLicense plateVIN...That's ... car catalog of some sort

Application UI

Product UI

Coordinates

click(134,32) //selects some record

click(215,122) //hits Properties

sleep(5) //sleeps to let dialog be painted

click(64,182) //expands color combo

click(235,182) //selects Gray

click(235,212) //hit OK

Td ~= 1.1 * Tm, Ts ~= 1 * Tm

Never tried, but ...

Widgets

Find Car records frame

Find table

Select 1abc234 cell

Push Properties button

Wait for 1abc234 dialog

Select Gray color in combo box

Push OK

Product UIDomainmodel

Car recordColorModelMakeYearLicense plateVIN

TestWidgets or coordinates

TD ~= 3 * TM, TS ~= .5 * TM

The values of course depend on how the fast product is changingThe estimations are for NetBeans we went throughthis stage

Now would happen if the combobox is replaced by color chooserpage up test would fail, 'cause it's looking for combobox page down

Most importantly ... all tests would fail!

UI Primitives

Find car list frame

CarListFrame list = new CarListFrame()Open properties dialog for car 1abc234

CarDialog propDialog = list.carProperties(1abc234);Set color to gray

propDialog.setColor(Color.GRAY);Apply changes

propDialog.ok();

Obvious solution

Product UIDomainmodel

Car record

Test library

ColorModelMakeYearLicense plateVINCarListFrameCarDialog

TestTD ~= 5 * TM, TS ~= .2 * TM

Domain model

Set color to gray for a car 1abc234

new CarRecord(1abc234). setColor(Color.GRAY);Underneath the cover, CarRecord class does all described earlier

ProductUIDomainmodel

Car record

Domain test libraryColorModelMakeYearLicense plateVINUI test libraryCarListCarDialog

CarRecord

TestTD ~= 7.5 * TM, TS ~= .05 * TM

Td and Ts together

TD and TS for NC=3, NR=8, TM=1

EA for NC=3, NR=8

Testing JavaFX UI
Tips from JavaFX quality team

Alexandre (Shura) IlineJava and JavaFX quality architectOracle

Links

http://jemmy.dev.java.net

++

Td/TmTs/Tm

Coordinates1.11

Widgets30.5

UI Library50.1

Domain library7.50.05

TdTsTd+(Ts*Nc*Nr)

Coordinates1.12425.1

Widgets31215

UI Library52.47.4

Domain library7.51.28.7

Ea

Coordinates0.9561752988

Widgets1.6

UI Library3.24324324324

Domain library2.75862068966

Nc = 1RatioTimeAutoManualEffect

1510.2

25.120.392

35.230.576

45.340.754

55.450.925

65.561.09

75.671.25

85.781.403

Nc = 3RatioTimeAutoManualEffect

1530.6

25.361.132

35.691.607

45.9122.033

56.2152.419

66.5182.769

76.8213.088

87.1233.239