Automation with Sikuli

AgendaWhat you’ll learn today

What is Sikuli?

How it works?

How to use it?

What is Sikuli ?

Visual approach to search and automation of graphical user interfaces using screenshots.

Motivation

Usually needed to automate GUIs: support from developers API access language/OS dependency position/naming dependencies

Just to see if it can be done

Demo

System design

Finding GUI patterns on the screen

Invariance

Resized versions

Texturally similar, different color pallets

Template Matching for small patterns

Invariant local features for big patterns

Learn Extract model from training pattern▪ Invariant to scale & rotation

Encode in the model the relative position of the center

Search Extract invariant features Infer possible model center position Cluster consistent features Validate supposition

Find visual objects

find( )

Finder

f = Finder(path-to-imagefile)

f.find(path-to-imagefile, [similarity])

// iterates through Match objectswhile(f.hasNext): print “found match: “ + f.next().getScore()

Pattern

Abstraction for visual patterns Are used by finding operations Methods can be chained to refine the

pattern Can define click points

Pattern(string)

Pattern(string).similar(0.9).targetOffset(10,30)

Region

Region(x, y, w, h)Region(region)Region(Rectangle)

Search in a given region Observe a region in background for changes Retrieve matches Optimize the search by chaining regions No need to be aware of the content

Act on visual objects

click( )

Actions

click(PSMLR), doubleClick(PSMLR)

dragDrop(PSMLR target, PSMLR destination)

Actions

Uses the Tesseract OCR engine

type(PSMLR, text)

text()

Actions

They can be used along with KeyModifiers

mouseDown(button), mouseUp(button)

keyDown(keys), keyUp(keys)

Actions

wait(PS), waitVanish(PS)

Actions

onAppear(PS, handler)

onVanish(PS, handler)

onChange([minChengedSize], handler)

Performance

A typical call to find() for a 100x100 target on a 1600x1200 screen takes less than 200 msec

Improve your searches

How performant do you want it to be?

Extensions

Record-playback Sikuli Guide …

Why not give it a try and make your own?

Limitations

Ok, the moment of truth!

Screenshots – unstable interfaces Visibility constraints

A paradigm shift requires a thinking shift

DemoWhat we’ve made of it