Selenium RCoff of your desktopand onto the grid

Jennifer Bevan & Jason Huggins{jbevan, hugs}@google.com

Google, Inc.August 24, 2007

2

What we have to share…

• Experience report on extending Selenium RC from desktopuse to running within a grid.

• Demo: “gridified” (© jason) Selenium RC for the masses.

– You don’t have to be Google to parallelize your selenium tests

• Future directions & addressing needs.

3

Background

• Selenium RC (org.openqa.selenium) uses javascript to interactwith browsers.

– Provides ability to inspect contents of pages programmatically.

– Executes only within the scope of a browser session.

• Can’t handle all modal dialog windows:

– File upload

– File download

– Browser crash report

– Supports injection “of large chunks of badly written javascript”[S. Stewart, GTAC’07] into the page.

• Google has created a “farm” of Selenium RC machines thatfacilitates parallelized testing of multiple configurations.

4

High-level Architecture

5

A couple of usage statistics…

• Over 10 projects’automated/continuous test systems,and many(uncounted)individual users.

• Gmail tests,running at 1thread/test, wentfrom taking over40 minutes to 3.5minutes

6

Experience Report

• We discovered many different issues while deploying andduring adoption of the Google Selenium RC farm.

– Browser/OS variances in configuration and capabilities

– Reliability and scalability issues

– Limitations that are out of the execution context of SeleniumRC

• Changes we make to Selenium RC to support our needs getpropagated to the open source repository.

7

Browser/OS Issues

• Top issue: test isolation

– Firefox: RC generates a user profile specific to a test session.

– IE: RC modifies registry settings and configures the LANconnection settings directly.

• Therefore, either IE tests must be externally isolated(individual VMs, single-tracked, etc.) or RC needs to manageaccess to the shared resources.

– The latter approach not used in current RC code.

8

Browser/OS Issues (cont.)

• Javascript evaluation within the browser

– Firefox (*chrome) allows tests to bypass security checks inspecific situations

– IE (*iehta) is expected to perform similarly, but isexperimental within Selenium RC.

• Therefore, we’ve only deployed *iexplore

• Basic tests (no https, etc.) are not a problem.

• Complex tests that work in *chrome are not immediatelyusable on *iehta.

• Limits ability to immediately test multiple configurations(requires test modification)

9

Reliability Issues

• When we initially deployed our Selenium RC farm, thedeadlock in the current RC code was not known.

– Concurrently, ongoing work on RC increased deadlockrate, resulting in a wider awareness.

• Is not related to browser type or test complexity, so allusers experience spurious test failures.

– Defensive strategy: retry failed tests

• Google and the Selenium RC development team areactively addressing this problem (top priority).

10

Reliability Issues (cont.)

• Selenium RC has a memory leak and a connection leak.

• We adopted a two-prong approach for this.

– Defensive strategy: periodically drain off tests and restart RC

– Offensive strategy: search and destroy source of leaks.

• Defensive strategy works, so for right now this is not our toppriority.

11

Reliability Issues (more)

• Current Selenium RC regression test suite contains bothfunctional and unit tests.

– And yet, most of them are “happy path” (© patrick) tests.

– All of the issues we’ve found were in code that passed theRC regression tests.

– Resulting uncertainty affects ability to quickly deploy newversions (with new RC features) on the farm.

• We are contributing tests to the open source repositorythat better exercises our new code and the surroundingpre-existing code.

12

Scalability Issues

• Session identification

– Current RC code uses a method of identifying sessions that isbased solely on time.

• For our purposes, this was not unique enough.

• Patch to improve uniqueness made, will be given to opensource repository.

• Multiple tests per RC instance

– Unofficial assumption by RC development team that eachsession is in an isolated VM.

• We are considering tradeoffs between this model and addingfull support for concurrent tests in one machine.

13

What about Performance?

• Sure -- making RC faster would be great.

– We’re focusing on fixes that affect RC’s “gridability”: reliability andscalability

– Also, the point of creating a grid is to make all of your tests run inthe amount of time it takes your longest test to run.

• At which point performance fixes get more attention.

14

Other contributions…

• Exposed bug in handling of InterruptedException withinSelenium RC.

– A waitFor…(elementName, 45000) statement “timed out” inunder 4 seconds, not 45 seconds.

– Added fix and tests for timeout calculation.

• Exposed tendency of Selenium RC to leave browsers openafter test session has ended.

– Usually when a test enters a deadlock or when a javascript “eval”stops responding.

– Current effort focused on eliminating deadlock; followup work willstrengthen browser shutdown method.

15

Out-of-scope RC limitations

• A grid of Selenium RC machines is not something you want tomaintain manually.

– So if IE “encounters an error” and gets wedged to the pointwhere no new IE session can be started…” -- do you reallywant to VNC in and click “Don’t Send” to clear the IE state?

• Out-of-band communication with a farm manager to handleplatform-specific browser error handling is necessary for large-scale deployment.

– And once you have the requisite ‘watchdog’ client on the RCmachine, you can also use it for metric collection, managingconfiguration recovery, etc…

16

(and now we switch to Jason…)

17

Which is faster? (1 server, sequential tasks)

Time

Servers

1

2

3

4

18

Which is faster? (4 servers, parallel tasks)

Time

Servers

1

2

3

4

19

How do you add more servers?

Thank this dude

20

Make computing a true utility (for anyone)

21

Pricing is cheap, but it’s not free.

$.10 per hour

==

~$74 per month

22

Demo…

23

Questions?

1. Are you going to open-source this?

2. Who is Paul Hammant?(http://paulhammant.com/)

24

Creator of Selenium Driven aka “Remote Control”

25

Thank you, Paul. :-)