Using Selenium to Improve a Team's Development Cycle

Selenium Conference 2012Mike Davis - Google

Background

Background

What is a Software Engineer in Test?

"Engineer who does everything and anything to improve productivity and quality of Google products."

How Can Selenium Improve a Team's Development Cycle?

● Catch bugs before submission

● Provide high confidence that system is working

● Release faster & more often

● Catch bugs not likely to be caught by manual QA

The Objectives

● Tests are written at the same time as features

● Tests are run before every check-in

● Code that breaks a test is not submitted

● Tests run at all stages of the development process

How Can Selenium Improve a Team's Development Cycle?

The Strategy

● Tests must be run as much as possible

● Developers must be involved in writing and maintaining tests

● We need really good tests

How Can Selenium Improve a Team's Development Cycle?

Some Corollaries

● Fast & Stable

● Easy to run

● Easy to read

● Easy to debug

● Easy to write

What Makes Good Functional Tests?

Tests Must Run From a Single Command (Or IDE)

● No "follow steps in this wiki"● No "first install this software"● No "Add yourself to this unix group"● No "First visit website and sign up a new user"● No command line flags

What Makes Good Functional Tests?

Run this command & everything "just works"

Hermetic Tests

Hermetic:● Self contained● No external dependencies

Managing dependencies:

● Run them in-memory as well● Look for in-memory implementations of data-stores● Fake out dependencies that can't be brought up

What Makes Good Functional Tests?

Keep Tests Small / Limit Number of Tests

Limit the number of tests:

● Do you really need this test?● Is this already covered by unit tests?● Can you write a unit test for this?

Keep tests small:

● Can you make it 2 tests?

What Makes Good Functional Tests?

Tests Set Up All Data

What Makes Good Functional Tests?

public void testUserHasOnePurchase() { AccountPage accountPage = loginPage.login("testUser1", "testPassword"); assertEquals(1, accountPage.getPurchaseCount());}

Example Failure 1:

Exception: Expected <1> got <2> Example Failure 2:

Exception: Login Failed

Errors Are Easy to Track Down

● Assertions have useful error messages

Exception : False line:225

Exception : Expected User to Have Purchase

VS

What Makes Good Functional Tests?

● Supply Screenshots of Error States

Exception: Timed out locating element "By.id = SearchButton". A screenshot has been saved at http://screenshot/_Ghktyc

Screenshot:

Errors Are Easy to Track Down

Exception: Timed out locating element "By.id = BuyButton"

● Misleading exceptions should be wrapped.

Exception: Error page detected locating element "By.id = BuyButton"

● Grab server side logsException: Error page detected locating element "By.id = BuyButton" ServerSide: NullPointerException @Purchase.java line 225

What Makes Good Functional Tests?

Examplepublic class OrdersTest extends TestCase {... public void setUp() { server = new LocalRunningServer(); server.initialize(); pathFinder = new PathFinder(server.getAddress()); } public void testOrdersShowInAccount() { User user = userCreator.createNewUser(); purchaser.makePurchaseForUser(user); HomePage homePage = pathFinder.getHomePage(); LoginPage loginPage = homePage.navigateToLoginPage(); UserDashboard userDashboard = loginPage.loginAs(user); OrdersPage ordersPage = userDashboard.navigateToMyOrders(); assertEquals(1, ordersPage.getNumberOfOrders()); }}

What Makes Good Functional Tests?

Counter Example

public class OrdersPage implements PageObject {... public WebElement getOrder() { if (!hasOrders()) { createOrder(); } return driver.findElement("order"); } public List<WebElement> getOrders() { return driver.findElements(By.name("order")); }}

public class OrdersPage implements PageObject {... public String getOrderId() { assert(hasOrders(), "Expected orders but none found") ; return driver.findElement("order").getText(); } public List<String> getOrderIds() { ... }}

Bad Better

What Makes Good Functional Tests?

Going Beyond

Testing In-Memory Servers vs Deployed Servers

Objective:● Run tests at all stages of development cycle

Server

Local Server

Remote Server

Going Beyond

Problem:● Priorities for Release Candidate are different● Already have good suite of tests, don't want to

write another

Profiling Tests

Key: GetHomePageData GetSpecialOffers GetUserData AddToBasket

History of Project

No. of Calls

TestLoadHomePage() - Call Profile at Every Commit

Browser ServerAjax

3

2

1

Going Beyond

Using Selenium to Improve a Team's Development Cycle:

● Run tests as often as possible● Get dev involved● Good tests

○ Fast & Stable○ Easy to run○ Easy to read○ Easy to debug○ Easy to write

There are loads of ways to extend a good suite of tests.

Summary

Questions?