wojciech seliga wojciech.seliga@spartez.com, t:...

5/16/12

How to survive automated tests hell

Wojciech Seligawojciech.seliga@spartez.com, T: @wseliga

1

Taming automated testsbeast

2

About me

• 29 years of coding

• Agile Practices (inc. TDD) since 2003

• ScrumMaster, Agile Coach, Trainer, Speaker

• 4.5 years with Atlassian (JIRA Development Team Lead)

• Co-founder of Spartez

3

The Story

4

Codebase10 years old

5

Growing from 2 to about 40 engineers

6

Obsessed with Quality

7

Obsessed with Automated Tests

8

... From the Beginning

9

1.5M lines of code

10

1.5M lines of code

10

1.5M lines of codeCheating

10

Mixture of technologies

11

Mixture of technologies

Java

JSP

Velocity

Webwork

OSGi

Spring

Pico

Soy

OfBiz EntityEngine

QuartzOS Workflow

Jersey

LESS

SeraphJDBC

LuceneJackson

Maven2

JQuery

Backbone.js SpringDM

XML

Active Objects

OAuth

OpenSocial

Javamail

Underscore

REST XStream

Guava

11

DependenciesLots of

12

Dependencies554

mvn dependency:list -DincludeScope=compile -o | grep :jar | cut -c 11- | sed s/:provided// | sed s/:compile//| sort -u|wc -l12

65 modules in one IntelliJ project13

65 modules in one IntelliJ project13

65 modules in one IntelliJ project13

14

13000 unit tests

15

Almost 1000 Selenium Tests

16

4000 Functional and Integration Tests

17

18

Atlassian JIRA

18

Our CI environment

19

Test frameworks

• JUnit 3 and 4

• JMock, Easymock, Mockito

• Powermock, Hamcrest

• QUnit, HtmlUnit

• JWebUnit, Selenium, WebDriver

• Custom runners, extensions, matchers

20

Test frameworks

• JUnit 3 and 4

• JMock, Easymock, Mockito

• Powermock, Hamcrest

• QUnit, HtmlUnit

• JWebUnit, Selenium, WebDriver

• Custom runners, extensions, matchers

20

Bamboo Setup

• Dedicated server with 60+ remote agents (including Amazon Elastic)

• Build engineers

• Bamboo devs

21

22

for each main branch22

for each main branch22

23

Run first

23

Run in parallel in batches

Run first

23

There is

MuchMore

24

Type of Tests

• Unit

• Functional

• Selenium / WebDriver

• Integration

• Platform

• Performance

25

Platforms

26

Platforms

• Dimension - DB: MySQL, PostgreSQL, MS SQL, Oracle

26

Platforms

• Dimension - DB: MySQL, PostgreSQL, MS SQL, Oracle

• Dimension - OS: Linux, Windows

26

Platforms

• Dimension - DB: MySQL, PostgreSQL, MS SQL, Oracle

• Dimension - OS: Linux, Windows

• Dimension - Java ver.: 1.5, 1.6, 1.7

26

Platforms

• Dimension - DB: MySQL, PostgreSQL, MS SQL, Oracle

• Dimension - OS: Linux, Windows

• Dimension - Java ver.: 1.5, 1.6, 1.7

• Dimension - CPU arch.: 32-bit, 64-bit

26

Platforms

• Dimension - DB: MySQL, PostgreSQL, MS SQL, Oracle

• Dimension - OS: Linux, Windows

• Dimension - Java ver.: 1.5, 1.6, 1.7

• Dimension - CPU arch.: 32-bit, 64-bit

• Dimension - Deployment Mode: Standalone, Tomcat, Websphere, Weblogic

26

Platforms

• Dimension - DB: MySQL, PostgreSQL, MS SQL, Oracle

• Dimension - OS: Linux, Windows

• Dimension - Java ver.: 1.5, 1.6, 1.7

• Dimension - CPU arch.: 32-bit, 64-bit

• Dimension - Deployment Mode: Standalone, Tomcat, Websphere, Weblogic

• Dimension - Browsers: IE 8+, FF, Chrome,

26

Platforms

• Dimension - DB: MySQL, PostgreSQL, MS SQL, Oracle

• Dimension - OS: Linux, Windows

• Dimension - Java ver.: 1.5, 1.6, 1.7

• Dimension - CPU arch.: 32-bit, 64-bit

• Dimension - Deployment Mode: Standalone, Tomcat, Websphere, Weblogic

• Dimension - Browsers: IE 8+, FF, Chrome,

26

Platforms

• Dimension - DB: MySQL, PostgreSQL, MS SQL, Oracle

• Dimension - OS: Linux, Windows

• Dimension - Java ver.: 1.5, 1.6, 1.7

• Dimension - CPU arch.: 32-bit, 64-bit

• Dimension - Deployment Mode: Standalone, Tomcat, Websphere, Weblogic

• Dimension - Browsers: IE 8+, FF, Chrome,

Coming

26

Platforms

• Dimension - DB: MySQL, PostgreSQL, MS SQL, Oracle

• Dimension - OS: Linux, Windows

• Dimension - Java ver.: 1.5, 1.6, 1.7

• Dimension - CPU arch.: 32-bit, 64-bit

• Dimension - Deployment Mode: Standalone, Tomcat, Websphere, Weblogic

• Dimension - Browsers: IE 8+, FF, Chrome,

Run Nightlyor Before Release

Coming

26

Triggering Builds

• On Commit (hooks, polling)

• Dependent Builds

• Nightly Builds

• Manual Builds

27

But...

28

Slow unit test29

Very slow functional tests30

Builds Wait in The Queue31

Builds Often Fail32

Too Often...33

It takes time to fix it...34

Sometimes very long35

You commit at 3 PM

36

You commit at 3 PM

You get “Unit Test Green” email at 4PM

36

You commit at 3 PM

You get “Unit Test Green” email at 4PM

You happily go home

36

You commit at 3 PM

You get “Unit Test Green” email at 4PM

You get flood of “Red Test X” emails at 4 - 9PM

You happily go home

36

You commit at 3 PM

You get “Unit Test Green” email at 4PM

You get flood of “Red Test X” emails at 4 - 9PM

Your colleagues on the other side of the globe

You happily go home

36

You commit at 3 PM

You get “Unit Test Green” email at 4PM

You get flood of “Red Test X” emails at 4 - 9PM

Your colleagues on the other side of the globe

You happily go home

You36

“Slow CI loop and non-deterministic tests are

strong inhibitor of change instead of the catalyst”

by W. Seliga37

“We probably spend more time dealing with the JIRA

test codebase than the production codebase”

38

0

25

50

75

100

Test Coverage

Effort Invested

Striving for Coverage

39

0%

25%

50%

75%

100%

Value

Size of automated tests

Strange? Relationship

40

Test Hell - The Outcomes

• Development slows down

• Devs are afraid of change

• Software difficult to release

• Significant amount of time spent on analysing test failures

• Morale goes down

41

FeedbackSpeed

`Quality

42

Feedback Loop Speed

• Tiniest change triggers test avalanche

• Lack of responsibility syndrome

• Devs do not run tests locally (speed)

• Before you get the results you are at home

43

Quality

• Non-deterministic tests (races, timeouts)

• Catching up with UI changes

• 1 red test hides new failures

• Ignoring always red tests in dangerous ...

44

Broken window theory45

Decisions which do not scale

46

Decisions which do not scale

• All unit tests in one maven module

46

Decisions which do not scale

• All unit tests in one maven module

• All functional tests in one maven module

46

Decisions which do not scale

• All unit tests in one maven module

• All functional tests in one maven module

• All Selenium and web-driver tests in one module

46

Decisions which do not scale

• All unit tests in one maven module

• All functional tests in one maven module

• All Selenium and web-driver tests in one module

• Every commit triggers rebuild and re-test of everything

46

Decisions which do not scale

• All unit tests in one maven module

• All functional tests in one maven module

• All Selenium and web-driver tests in one module

• Every commit triggers rebuild and re-test of everything

• Monolithic test framework / utils

46

Decisions which do not scale

• All unit tests in one maven module

• All functional tests in one maven module

• All Selenium and web-driver tests in one module

• Every commit triggers rebuild and re-test of everything

• Monolithic test framework / utils

• Opaque fixtures

46

Strategies

47

Test Quality

48

Catching up with UI changes

Problem:

49

Catching up with UI changes

Problem:

Solution:

49

Catching up with UI changes

Page Objects Pattern

Problem:

Solution:

49

Page Objects Pattern• Page Objects model UI elements (pages,

components, dialogs, areas) your tests interact with

• Page Objects shield tests from changing internal structure of the page

• Page Objects generally do not make assertions about data. The can assert the state.

• Designed for chaining50

Page Objects Examplepublic class AddUserPage extends AbstractJiraPage{

private static final String URI = "/secure/admin/user/AddUser!default.jspa";

@ElementBy(name = "username") private PageElement username;

@ElementBy(name = "password") private PageElement password;

@ElementBy(name = "confirm") private PageElement passwordConfirmation;

@ElementBy(name = "fullname") private PageElement fullName;

@ElementBy(name = "email") private PageElement email;

@ElementBy(name = "sendemail") private PageElement sendEmail;

@ElementBy(id = "user-create-submit") private PageElement submit;

@ElementBy (id = "user-create-cancel") private PageElement cancelButton;

@Override public String getUrl() { return URI; }

...

@Override public TimedCondition isAt() { return and(username.timed().isPresent(), password.timed().isPresent(), fullName.timed().isPresent()); }

public AddUserPage addUser(final String username, final String password, final String fullName, final String email, final boolean receiveEmail)

{ this.username.type(username); this.password.type(password); this.passwordConfirmation.type(password); this.fullName.type(fullName); this.email.type(email); if(receiveEmail) { this.sendEmail.select(); } return this; }

public ViewUserPage createUser() { return createUser(ViewUserPage.class); }

public <T extends Page> T createUser(Class<T> nextPage, Object...args) { submit.click(); return pageBinder.bind(nextPage, args); }

51

Using Page Objects @Test public void testServerError() { jira.gotoLoginPage().loginAsSysAdmin(AddUserPage.class) .addUser("username", "mypassword", "My Name",

"sample@email.com", false) .createUser();

// assertions here }

52

Using Page Objects @Test public void testImportSampleProject() { final PivotalImporterSetupPage setupPage = getSetupPage(); assertEquals("1. Connect", setupPage.getActiveTabText());

final PivotalProjectsMappingsPage projectMappingPage = setupPage.next(); assertEquals("2. Project Mapping", setupPage.getActiveTabText()); assertTrue("Expecting all project to be selected by default", projectMappingPage.areAllProjectsSelected()); projectMappingPage.setImportAllProjects(false); projectMappingPage.setProjectImported(sampleProject, true); projectMappingPage.createProject(sampleProject, sampleProject, "SAMPLE");

final ImporterFinishedPage importerLogsPage = projectMappingPage.beginImport().waitUntilFinished(); assertTrue(importerLogsPage.isSuccess()); assertEquals(0, importerLogsPage.getGlobalErrors().size()); assertEquals("1", importerLogsPage.getProjectsImported()); // more assertions here }

53

More on Page Objects

• Design for reusability

• Design for sharing - libraries of Page Objects

• Good support by WebDriver/Selenium 2

• Atlassian Selenium 2.0

54

Page Objects Caveat

The easier is to write functional tests, the more tests will be written...

The more stuff you will need to maintain,the more time the execution of your tests will take...

W. Seliga, PO. 1-4

55

Opaque Test FixturesProblem:

56

Opaque Test FixturesProblem:

Solution:

56

Opaque Test Fixtures

REST-based Set-up,Backdoors

Problem:

Solution:

56

REST-based Setup

57

REST-based Setup @Before public void setUpTest() { restore("some-big-xml-file-with-everything-needed-inside.xml"); }

57

REST-based Setup @Before public void setUpTest() { restore("some-big-xml-file-with-everything-needed-inside.xml"); }

VS

57

REST-based Setup @Before public void setUpTest() { restore("some-big-xml-file-with-everything-needed-inside.xml"); }

@Before public void setUpTest() { restClient.restoreEmptyInstance(); restClient.createProject(/* project params */); restClient.createUser(/* user params */); restClient.createUser(/* user params */); restClient.createSomethingElse(/* ... */); }

VS

57

Flakey TestsProblem:

58

Flakey TestsProblem:

Solution:

58

Flakey Tests

Quarantine

Problem:

Solution:

58

Flakey Tests

Quarantine

Problem:

Solution:

Fix58

Flakey Tests

Quarantine

Problem:

Solution:

Fix Eradicate58

Quarantine• @Ignore, @Category

• Quarantine on CI server

• Recover or Die - limiting quarantine

59

Quarantine• @Ignore, @Category

• Quarantine on CI server

• Recover or Die - limiting quarantine

59

Quarantine• @Ignore, @Category

• Quarantine on CI server

• Recover or Die - limiting quarantine

59

Heal or Kill

60

Heal or Kill

60

Quickly!

Fixing Flakey TestsProblem:

61

Fixing Flakey TestsProblem:

Solution:

61

Fixing Flakey Tests

Timed Conditions

Problem:

Solution:

61

Fixing Flakey Tests

Timed Conditions

Problem:

Solution:

Test-friendly Markup61

Fixing Flakey Tests

Timed Conditions

Problem:

Solution:

Mock Unreliable DepsTest-friendly Markup

61

Timed Conditions

62

Test-friendly Markup

• Do not save on IDs

• Do not save on CSS classes

• XPath is fragile

• XPath is expensive

• XPath is not readable

• i18N

63

Mock Unreliable Dependencies

64

Speed

65

SpeedAiming at 10 seconds build

65

Splitting CodebaseThe easiest and most effective

improvement

66

Splitting Codebase

• Tests closer to tested code

• Less to test

• Testing less frequently

• Increased team responsibility

• Restructuring CI hierarchy - more complicated picture

67

Less to Test

68

Less to TestCommit

68

Less to TestCommit

68

Less to TestCommit

68

Less to TestCommit

68

Less to Test

68

Less to Test

Most of commits happen here68

Speed vs Control Workspace Dilemma

• Incubation

• Maturity

• Custom workspaces

69

Test Execution Time

70

Execution Time:Test Level

71

Execution Time:Test Level

Unit Tests

71

Execution Time:Test Level

Unit Tests

REST API Tests

71

Execution Time:Test Level

Unit Tests

REST API Tests

JWebUnit/HTMLUnit Tests

71

Execution Time:Test Level

Unit Tests

REST API Tests

JWebUnit/HTMLUnit Tests

Selenium/WebDriver Tests

71

Execution Time:Test Level

Unit Tests

REST API Tests

JWebUnit/HTMLUnit Tests

Selenium/WebDriver Tests

71

Execution Time:Test Level

Unit Tests

REST API Tests

JWebUnit/HTMLUnit Tests

Selenium/WebDriver Tests

Speed

71

Execution Time:Test Level

Unit Tests

REST API Tests

JWebUnit/HTMLUnit Tests

Selenium/WebDriver Tests

Speed

71

Execution Time:Test Level

Unit Tests

REST API Tests

JWebUnit/HTMLUnit Tests

Selenium/WebDriver Tests

Speed Confidence

71

Test Pyramid

Unit Tests

REST / HTML Tests

Selenium

72

Test Pyramid

Unit Tests

REST / HTML Tests

Selenium

72

Execution time - Cont.

• Batching

• Several tests per single set-up (violation of test isolation)

• REST-based setups and assertions

• Remove / merge overlapping tests

73

Execution time - Cont.

• IDs over CSS/JQuery Selectors over XPath

• JUnit tests running in the container

• In-process testing

• In-memory DBs

• Mocking web servers

• Reducing framework initialization time

• Test Optimization (Clover)

74

Waiting time

• More build agents

• Shorter and smaller tests

• No sleep()

• Avoiding long fixture setup (hot container, fragmented setup)

75

Preparation Time

• SCM performance

• Container set-up

• Compilation time (GWT...)

• Maven...

• Artifacts passing

76

So how about the goals?

Is 10 seconds build realistic?

77

Realistic Goals (for us*)

78

Realistic Goals (for us*)

*My current personal dreams78

Realistic Goals (for us*)

Time Type

2 min for unit tests for 95% of the commits

5 min for base smoke functional tests for 95% of the commits

15 min for ALL tests for 95% of the commits on selected platform

30 min for ALL tests for ALL commits

*My current personal dreams78

Realistic Goals (for us*)

Time Type

2 min for unit tests for 95% of the commits

5 min for base smoke functional tests for 95% of the commits

15 min for ALL tests for 95% of the commits on selected platform

30 min for ALL tests for ALL commits

*My current personal dreams78

Realistic Goals (for us*)

Time Type

2 min for unit tests for 95% of the commits

5 min for base smoke functional tests for 95% of the commits

15 min for ALL tests for 95% of the commits on selected platform

30 min for ALL tests for ALL commits

*My current personal dreams78

Realistic Goals (for us*)

Time Type

2 min for unit tests for 95% of the commits

5 min for base smoke functional tests for 95% of the commits

15 min for ALL tests for 95% of the commits on selected platform

30 min for ALL tests for ALL commits

*My current personal dreams78

Realistic Goals (for us*)

Time Type

2 min for unit tests for 95% of the commits

5 min for base smoke functional tests for 95% of the commits

15 min for ALL tests for 95% of the commits on selected platform

30 min for ALL tests for ALL commits

*My current personal dreams78

Realistic Goals (for us*) p.2

79

Realistic Goals (for us*) p.2

*My current personal dreams79

Realistic Goals (for us*) p.2

Metric Type

>98% green unit tests

>95% green functional tests

<20min average time to fix unit test

<2h average time to fix functional test

*My current personal dreams79

Realistic Goals (for us*) p.2

Metric Type

>98% green unit tests

>95% green functional tests

<20min average time to fix unit test

<2h average time to fix functional test

*My current personal dreams79

Realistic Goals (for us*) p.2

Metric Type

>98% green unit tests

>95% green functional tests

<20min average time to fix unit test

<2h average time to fix functional test

*My current personal dreams79

Realistic Goals (for us*) p.2

Metric Type

>98% green unit tests

>95% green functional tests

<20min average time to fix unit test

<2h average time to fix functional test

*My current personal dreams79

Realistic Goals (for us*) p.2

Metric Type

>98% green unit tests

>95% green functional tests

<20min average time to fix unit test

<2h average time to fix functional test

*My current personal dreams79

Our Possible Future

• Further splitting the code-base, incubation and maturity

• Finer-grained team responsibilities

• Merciless quarantine and purging of flakey tests

• More page objects, less old-school Selenium

• Refactoring/removal of slow tests

• More REST-driven test fixtures and assertions

• More developer's awareness, training, coaching

80

Take-aways

81

Automated testing has cumulative benefits

82

Automated testing has cumulative benefits

...and cumulative cost

82

Splitting codebase and favouring unit tests

are key aspects of short test feedback loop

83

Test Code is Not Trash

84

Test Code is Not Trash

Respect

84

Test Code is Not Trash

DesignRespect

84

Test Code is Not Trash

Design

Maintain

Respect

84

Test Code is Not Trash

Design

MaintainReview

Respect

84

Test Code is Not Trash

Design

MaintainRefactor

Review

Respect

84

Test Code is Not Trash

Design

MaintainRefactor

Review

Respect Restructure

84

Test Code is Not Trash

Design

MaintainRefactor

Share

Review

Respect Restructure

84

Test Code is Not Trash

Design

MaintainRefactor

Share

Review

Respect

Discuss

Restructure

84

Test Code is Not Trash

Design

MaintainRefactor

Share

Review

Prune

Respect

Discuss

Restructure

84

Optimum Balance

85

Optimum Balance

Isolation85

Optimum Balance

Isolation Speed85

Optimum Balance

Isolation Speed Coverage85

Optimum Balance

Isolation Speed Coverage Level85

Optimum Balance

Isolation Speed Coverage Level Structure85

Optimum Balance

Isolation Speed Coverage Level Structure Effort85

Dangerous to temper with

86

Dangerous to temper with

Quality / Determinism86

Dangerous to temper with

MaintainabilityQuality / Determinism86

87

There are no universal rules - silver bullets

87

There are no universal rules - silver bullets

We are expected to find optimum balance for our

specific case

87

There are no universal rules - silver bullets

We are expected to find optimum balance for our

specific case

Definition of “optimum” constantly changes

87

Otherwise

88

Otherwise

89

Did I mention that Page Objects pattern?

90

We are hiring in Gdańsk!

Java devs Javascript devs UI/UX designers

Team leader

jobs@spartez.com

Do you want to help us?

91

Credits

• Photos:• http://www.flickr.com/photos/toofarnorth/ - Dragon

• http://www.flickr.com/photos/striatic - Frustration

• http://www.flickr.com/photos/leeadlaf/ - Broken window

• http://www.flickr.com/photos/johnloo/ - Lightbulb

• Dariusz Kordoński - for Test Improvements Leadership in JIRA

92

Thank You

93