![Page 2: Wojciech Seliga wojciech.seliga@spartez.com, T: @wseligas3-eu-west-1.amazonaws.com/presentations2012/36_presentation.pdf · 5/16/12 How to survive automated tests hell Wojciech Seliga](https://reader031.vdocument.in/reader031/viewer/2022013008/5e3af5ab1060d64d7023d879/html5/thumbnails/2.jpg)
Taming automated testsbeast
2
![Page 3: Wojciech Seliga wojciech.seliga@spartez.com, T: @wseligas3-eu-west-1.amazonaws.com/presentations2012/36_presentation.pdf · 5/16/12 How to survive automated tests hell Wojciech Seliga](https://reader031.vdocument.in/reader031/viewer/2022013008/5e3af5ab1060d64d7023d879/html5/thumbnails/3.jpg)
About me
• 29 years of coding
• Agile Practices (inc. TDD) since 2003
• ScrumMaster, Agile Coach, Trainer, Speaker
• 4.5 years with Atlassian (JIRA Development Team Lead)
• Co-founder of Spartez
3
![Page 4: Wojciech Seliga wojciech.seliga@spartez.com, T: @wseligas3-eu-west-1.amazonaws.com/presentations2012/36_presentation.pdf · 5/16/12 How to survive automated tests hell Wojciech Seliga](https://reader031.vdocument.in/reader031/viewer/2022013008/5e3af5ab1060d64d7023d879/html5/thumbnails/4.jpg)
The Story
4
![Page 5: Wojciech Seliga wojciech.seliga@spartez.com, T: @wseligas3-eu-west-1.amazonaws.com/presentations2012/36_presentation.pdf · 5/16/12 How to survive automated tests hell Wojciech Seliga](https://reader031.vdocument.in/reader031/viewer/2022013008/5e3af5ab1060d64d7023d879/html5/thumbnails/5.jpg)
Codebase10 years old
5
![Page 6: Wojciech Seliga wojciech.seliga@spartez.com, T: @wseligas3-eu-west-1.amazonaws.com/presentations2012/36_presentation.pdf · 5/16/12 How to survive automated tests hell Wojciech Seliga](https://reader031.vdocument.in/reader031/viewer/2022013008/5e3af5ab1060d64d7023d879/html5/thumbnails/6.jpg)
Growing from 2 to about 40 engineers
6
![Page 7: Wojciech Seliga wojciech.seliga@spartez.com, T: @wseligas3-eu-west-1.amazonaws.com/presentations2012/36_presentation.pdf · 5/16/12 How to survive automated tests hell Wojciech Seliga](https://reader031.vdocument.in/reader031/viewer/2022013008/5e3af5ab1060d64d7023d879/html5/thumbnails/7.jpg)
Obsessed with Quality
7
![Page 8: Wojciech Seliga wojciech.seliga@spartez.com, T: @wseligas3-eu-west-1.amazonaws.com/presentations2012/36_presentation.pdf · 5/16/12 How to survive automated tests hell Wojciech Seliga](https://reader031.vdocument.in/reader031/viewer/2022013008/5e3af5ab1060d64d7023d879/html5/thumbnails/8.jpg)
Obsessed with Automated Tests
8
![Page 9: Wojciech Seliga wojciech.seliga@spartez.com, T: @wseligas3-eu-west-1.amazonaws.com/presentations2012/36_presentation.pdf · 5/16/12 How to survive automated tests hell Wojciech Seliga](https://reader031.vdocument.in/reader031/viewer/2022013008/5e3af5ab1060d64d7023d879/html5/thumbnails/9.jpg)
... From the Beginning
9
![Page 10: Wojciech Seliga wojciech.seliga@spartez.com, T: @wseligas3-eu-west-1.amazonaws.com/presentations2012/36_presentation.pdf · 5/16/12 How to survive automated tests hell Wojciech Seliga](https://reader031.vdocument.in/reader031/viewer/2022013008/5e3af5ab1060d64d7023d879/html5/thumbnails/10.jpg)
1.5M lines of code
10
![Page 11: Wojciech Seliga wojciech.seliga@spartez.com, T: @wseligas3-eu-west-1.amazonaws.com/presentations2012/36_presentation.pdf · 5/16/12 How to survive automated tests hell Wojciech Seliga](https://reader031.vdocument.in/reader031/viewer/2022013008/5e3af5ab1060d64d7023d879/html5/thumbnails/11.jpg)
1.5M lines of code
10
![Page 12: Wojciech Seliga wojciech.seliga@spartez.com, T: @wseligas3-eu-west-1.amazonaws.com/presentations2012/36_presentation.pdf · 5/16/12 How to survive automated tests hell Wojciech Seliga](https://reader031.vdocument.in/reader031/viewer/2022013008/5e3af5ab1060d64d7023d879/html5/thumbnails/12.jpg)
1.5M lines of codeCheating
10
![Page 13: Wojciech Seliga wojciech.seliga@spartez.com, T: @wseligas3-eu-west-1.amazonaws.com/presentations2012/36_presentation.pdf · 5/16/12 How to survive automated tests hell Wojciech Seliga](https://reader031.vdocument.in/reader031/viewer/2022013008/5e3af5ab1060d64d7023d879/html5/thumbnails/13.jpg)
Mixture of technologies
11
![Page 14: Wojciech Seliga wojciech.seliga@spartez.com, T: @wseligas3-eu-west-1.amazonaws.com/presentations2012/36_presentation.pdf · 5/16/12 How to survive automated tests hell Wojciech Seliga](https://reader031.vdocument.in/reader031/viewer/2022013008/5e3af5ab1060d64d7023d879/html5/thumbnails/14.jpg)
Mixture of technologies
Java
JSP
Velocity
Webwork
OSGi
Spring
Pico
Soy
OfBiz EntityEngine
QuartzOS Workflow
Jersey
LESS
SeraphJDBC
LuceneJackson
Maven2
JQuery
Backbone.js SpringDM
XML
Active Objects
OAuth
OpenSocial
Javamail
Underscore
REST XStream
Guava
11
![Page 15: Wojciech Seliga wojciech.seliga@spartez.com, T: @wseligas3-eu-west-1.amazonaws.com/presentations2012/36_presentation.pdf · 5/16/12 How to survive automated tests hell Wojciech Seliga](https://reader031.vdocument.in/reader031/viewer/2022013008/5e3af5ab1060d64d7023d879/html5/thumbnails/15.jpg)
DependenciesLots of
12
![Page 16: Wojciech Seliga wojciech.seliga@spartez.com, T: @wseligas3-eu-west-1.amazonaws.com/presentations2012/36_presentation.pdf · 5/16/12 How to survive automated tests hell Wojciech Seliga](https://reader031.vdocument.in/reader031/viewer/2022013008/5e3af5ab1060d64d7023d879/html5/thumbnails/16.jpg)
Dependencies554
mvn dependency:list -DincludeScope=compile -o | grep :jar | cut -c 11- | sed s/:provided// | sed s/:compile//| sort -u|wc -l12
![Page 17: Wojciech Seliga wojciech.seliga@spartez.com, T: @wseligas3-eu-west-1.amazonaws.com/presentations2012/36_presentation.pdf · 5/16/12 How to survive automated tests hell Wojciech Seliga](https://reader031.vdocument.in/reader031/viewer/2022013008/5e3af5ab1060d64d7023d879/html5/thumbnails/17.jpg)
65 modules in one IntelliJ project13
![Page 18: Wojciech Seliga wojciech.seliga@spartez.com, T: @wseligas3-eu-west-1.amazonaws.com/presentations2012/36_presentation.pdf · 5/16/12 How to survive automated tests hell Wojciech Seliga](https://reader031.vdocument.in/reader031/viewer/2022013008/5e3af5ab1060d64d7023d879/html5/thumbnails/18.jpg)
65 modules in one IntelliJ project13
![Page 19: Wojciech Seliga wojciech.seliga@spartez.com, T: @wseligas3-eu-west-1.amazonaws.com/presentations2012/36_presentation.pdf · 5/16/12 How to survive automated tests hell Wojciech Seliga](https://reader031.vdocument.in/reader031/viewer/2022013008/5e3af5ab1060d64d7023d879/html5/thumbnails/19.jpg)
65 modules in one IntelliJ project13
![Page 20: Wojciech Seliga wojciech.seliga@spartez.com, T: @wseligas3-eu-west-1.amazonaws.com/presentations2012/36_presentation.pdf · 5/16/12 How to survive automated tests hell Wojciech Seliga](https://reader031.vdocument.in/reader031/viewer/2022013008/5e3af5ab1060d64d7023d879/html5/thumbnails/20.jpg)
14
![Page 21: Wojciech Seliga wojciech.seliga@spartez.com, T: @wseligas3-eu-west-1.amazonaws.com/presentations2012/36_presentation.pdf · 5/16/12 How to survive automated tests hell Wojciech Seliga](https://reader031.vdocument.in/reader031/viewer/2022013008/5e3af5ab1060d64d7023d879/html5/thumbnails/21.jpg)
13000 unit tests
15
![Page 22: Wojciech Seliga wojciech.seliga@spartez.com, T: @wseligas3-eu-west-1.amazonaws.com/presentations2012/36_presentation.pdf · 5/16/12 How to survive automated tests hell Wojciech Seliga](https://reader031.vdocument.in/reader031/viewer/2022013008/5e3af5ab1060d64d7023d879/html5/thumbnails/22.jpg)
Almost 1000 Selenium Tests
16
![Page 23: Wojciech Seliga wojciech.seliga@spartez.com, T: @wseligas3-eu-west-1.amazonaws.com/presentations2012/36_presentation.pdf · 5/16/12 How to survive automated tests hell Wojciech Seliga](https://reader031.vdocument.in/reader031/viewer/2022013008/5e3af5ab1060d64d7023d879/html5/thumbnails/23.jpg)
4000 Functional and Integration Tests
17
![Page 24: Wojciech Seliga wojciech.seliga@spartez.com, T: @wseligas3-eu-west-1.amazonaws.com/presentations2012/36_presentation.pdf · 5/16/12 How to survive automated tests hell Wojciech Seliga](https://reader031.vdocument.in/reader031/viewer/2022013008/5e3af5ab1060d64d7023d879/html5/thumbnails/24.jpg)
18
![Page 25: Wojciech Seliga wojciech.seliga@spartez.com, T: @wseligas3-eu-west-1.amazonaws.com/presentations2012/36_presentation.pdf · 5/16/12 How to survive automated tests hell Wojciech Seliga](https://reader031.vdocument.in/reader031/viewer/2022013008/5e3af5ab1060d64d7023d879/html5/thumbnails/25.jpg)
Atlassian JIRA
18
![Page 26: Wojciech Seliga wojciech.seliga@spartez.com, T: @wseligas3-eu-west-1.amazonaws.com/presentations2012/36_presentation.pdf · 5/16/12 How to survive automated tests hell Wojciech Seliga](https://reader031.vdocument.in/reader031/viewer/2022013008/5e3af5ab1060d64d7023d879/html5/thumbnails/26.jpg)
Our CI environment
19
![Page 27: Wojciech Seliga wojciech.seliga@spartez.com, T: @wseligas3-eu-west-1.amazonaws.com/presentations2012/36_presentation.pdf · 5/16/12 How to survive automated tests hell Wojciech Seliga](https://reader031.vdocument.in/reader031/viewer/2022013008/5e3af5ab1060d64d7023d879/html5/thumbnails/27.jpg)
Test frameworks
• JUnit 3 and 4
• JMock, Easymock, Mockito
• Powermock, Hamcrest
• QUnit, HtmlUnit
• JWebUnit, Selenium, WebDriver
• Custom runners, extensions, matchers
20
![Page 28: Wojciech Seliga wojciech.seliga@spartez.com, T: @wseligas3-eu-west-1.amazonaws.com/presentations2012/36_presentation.pdf · 5/16/12 How to survive automated tests hell Wojciech Seliga](https://reader031.vdocument.in/reader031/viewer/2022013008/5e3af5ab1060d64d7023d879/html5/thumbnails/28.jpg)
Test frameworks
• JUnit 3 and 4
• JMock, Easymock, Mockito
• Powermock, Hamcrest
• QUnit, HtmlUnit
• JWebUnit, Selenium, WebDriver
• Custom runners, extensions, matchers
20
![Page 29: Wojciech Seliga wojciech.seliga@spartez.com, T: @wseligas3-eu-west-1.amazonaws.com/presentations2012/36_presentation.pdf · 5/16/12 How to survive automated tests hell Wojciech Seliga](https://reader031.vdocument.in/reader031/viewer/2022013008/5e3af5ab1060d64d7023d879/html5/thumbnails/29.jpg)
Bamboo Setup
• Dedicated server with 60+ remote agents (including Amazon Elastic)
• Build engineers
• Bamboo devs
21
![Page 30: Wojciech Seliga wojciech.seliga@spartez.com, T: @wseligas3-eu-west-1.amazonaws.com/presentations2012/36_presentation.pdf · 5/16/12 How to survive automated tests hell Wojciech Seliga](https://reader031.vdocument.in/reader031/viewer/2022013008/5e3af5ab1060d64d7023d879/html5/thumbnails/30.jpg)
22
![Page 31: Wojciech Seliga wojciech.seliga@spartez.com, T: @wseligas3-eu-west-1.amazonaws.com/presentations2012/36_presentation.pdf · 5/16/12 How to survive automated tests hell Wojciech Seliga](https://reader031.vdocument.in/reader031/viewer/2022013008/5e3af5ab1060d64d7023d879/html5/thumbnails/31.jpg)
for each main branch22
![Page 32: Wojciech Seliga wojciech.seliga@spartez.com, T: @wseligas3-eu-west-1.amazonaws.com/presentations2012/36_presentation.pdf · 5/16/12 How to survive automated tests hell Wojciech Seliga](https://reader031.vdocument.in/reader031/viewer/2022013008/5e3af5ab1060d64d7023d879/html5/thumbnails/32.jpg)
for each main branch22
![Page 33: Wojciech Seliga wojciech.seliga@spartez.com, T: @wseligas3-eu-west-1.amazonaws.com/presentations2012/36_presentation.pdf · 5/16/12 How to survive automated tests hell Wojciech Seliga](https://reader031.vdocument.in/reader031/viewer/2022013008/5e3af5ab1060d64d7023d879/html5/thumbnails/33.jpg)
23
![Page 34: Wojciech Seliga wojciech.seliga@spartez.com, T: @wseligas3-eu-west-1.amazonaws.com/presentations2012/36_presentation.pdf · 5/16/12 How to survive automated tests hell Wojciech Seliga](https://reader031.vdocument.in/reader031/viewer/2022013008/5e3af5ab1060d64d7023d879/html5/thumbnails/34.jpg)
Run first
23
![Page 35: Wojciech Seliga wojciech.seliga@spartez.com, T: @wseligas3-eu-west-1.amazonaws.com/presentations2012/36_presentation.pdf · 5/16/12 How to survive automated tests hell Wojciech Seliga](https://reader031.vdocument.in/reader031/viewer/2022013008/5e3af5ab1060d64d7023d879/html5/thumbnails/35.jpg)
Run in parallel in batches
Run first
23
![Page 36: Wojciech Seliga wojciech.seliga@spartez.com, T: @wseligas3-eu-west-1.amazonaws.com/presentations2012/36_presentation.pdf · 5/16/12 How to survive automated tests hell Wojciech Seliga](https://reader031.vdocument.in/reader031/viewer/2022013008/5e3af5ab1060d64d7023d879/html5/thumbnails/36.jpg)
There is
MuchMore
24
![Page 37: Wojciech Seliga wojciech.seliga@spartez.com, T: @wseligas3-eu-west-1.amazonaws.com/presentations2012/36_presentation.pdf · 5/16/12 How to survive automated tests hell Wojciech Seliga](https://reader031.vdocument.in/reader031/viewer/2022013008/5e3af5ab1060d64d7023d879/html5/thumbnails/37.jpg)
Type of Tests
• Unit
• Functional
• Selenium / WebDriver
• Integration
• Platform
• Performance
25
![Page 38: Wojciech Seliga wojciech.seliga@spartez.com, T: @wseligas3-eu-west-1.amazonaws.com/presentations2012/36_presentation.pdf · 5/16/12 How to survive automated tests hell Wojciech Seliga](https://reader031.vdocument.in/reader031/viewer/2022013008/5e3af5ab1060d64d7023d879/html5/thumbnails/38.jpg)
Platforms
26
![Page 39: Wojciech Seliga wojciech.seliga@spartez.com, T: @wseligas3-eu-west-1.amazonaws.com/presentations2012/36_presentation.pdf · 5/16/12 How to survive automated tests hell Wojciech Seliga](https://reader031.vdocument.in/reader031/viewer/2022013008/5e3af5ab1060d64d7023d879/html5/thumbnails/39.jpg)
Platforms
• Dimension - DB: MySQL, PostgreSQL, MS SQL, Oracle
26
![Page 40: Wojciech Seliga wojciech.seliga@spartez.com, T: @wseligas3-eu-west-1.amazonaws.com/presentations2012/36_presentation.pdf · 5/16/12 How to survive automated tests hell Wojciech Seliga](https://reader031.vdocument.in/reader031/viewer/2022013008/5e3af5ab1060d64d7023d879/html5/thumbnails/40.jpg)
Platforms
• Dimension - DB: MySQL, PostgreSQL, MS SQL, Oracle
• Dimension - OS: Linux, Windows
26
![Page 41: Wojciech Seliga wojciech.seliga@spartez.com, T: @wseligas3-eu-west-1.amazonaws.com/presentations2012/36_presentation.pdf · 5/16/12 How to survive automated tests hell Wojciech Seliga](https://reader031.vdocument.in/reader031/viewer/2022013008/5e3af5ab1060d64d7023d879/html5/thumbnails/41.jpg)
Platforms
• Dimension - DB: MySQL, PostgreSQL, MS SQL, Oracle
• Dimension - OS: Linux, Windows
• Dimension - Java ver.: 1.5, 1.6, 1.7
26
![Page 42: Wojciech Seliga wojciech.seliga@spartez.com, T: @wseligas3-eu-west-1.amazonaws.com/presentations2012/36_presentation.pdf · 5/16/12 How to survive automated tests hell Wojciech Seliga](https://reader031.vdocument.in/reader031/viewer/2022013008/5e3af5ab1060d64d7023d879/html5/thumbnails/42.jpg)
Platforms
• Dimension - DB: MySQL, PostgreSQL, MS SQL, Oracle
• Dimension - OS: Linux, Windows
• Dimension - Java ver.: 1.5, 1.6, 1.7
• Dimension - CPU arch.: 32-bit, 64-bit
26
![Page 43: Wojciech Seliga wojciech.seliga@spartez.com, T: @wseligas3-eu-west-1.amazonaws.com/presentations2012/36_presentation.pdf · 5/16/12 How to survive automated tests hell Wojciech Seliga](https://reader031.vdocument.in/reader031/viewer/2022013008/5e3af5ab1060d64d7023d879/html5/thumbnails/43.jpg)
Platforms
• Dimension - DB: MySQL, PostgreSQL, MS SQL, Oracle
• Dimension - OS: Linux, Windows
• Dimension - Java ver.: 1.5, 1.6, 1.7
• Dimension - CPU arch.: 32-bit, 64-bit
• Dimension - Deployment Mode: Standalone, Tomcat, Websphere, Weblogic
26
![Page 44: Wojciech Seliga wojciech.seliga@spartez.com, T: @wseligas3-eu-west-1.amazonaws.com/presentations2012/36_presentation.pdf · 5/16/12 How to survive automated tests hell Wojciech Seliga](https://reader031.vdocument.in/reader031/viewer/2022013008/5e3af5ab1060d64d7023d879/html5/thumbnails/44.jpg)
Platforms
• Dimension - DB: MySQL, PostgreSQL, MS SQL, Oracle
• Dimension - OS: Linux, Windows
• Dimension - Java ver.: 1.5, 1.6, 1.7
• Dimension - CPU arch.: 32-bit, 64-bit
• Dimension - Deployment Mode: Standalone, Tomcat, Websphere, Weblogic
• Dimension - Browsers: IE 8+, FF, Chrome,
26
![Page 45: Wojciech Seliga wojciech.seliga@spartez.com, T: @wseligas3-eu-west-1.amazonaws.com/presentations2012/36_presentation.pdf · 5/16/12 How to survive automated tests hell Wojciech Seliga](https://reader031.vdocument.in/reader031/viewer/2022013008/5e3af5ab1060d64d7023d879/html5/thumbnails/45.jpg)
Platforms
• Dimension - DB: MySQL, PostgreSQL, MS SQL, Oracle
• Dimension - OS: Linux, Windows
• Dimension - Java ver.: 1.5, 1.6, 1.7
• Dimension - CPU arch.: 32-bit, 64-bit
• Dimension - Deployment Mode: Standalone, Tomcat, Websphere, Weblogic
• Dimension - Browsers: IE 8+, FF, Chrome,
26
![Page 46: Wojciech Seliga wojciech.seliga@spartez.com, T: @wseligas3-eu-west-1.amazonaws.com/presentations2012/36_presentation.pdf · 5/16/12 How to survive automated tests hell Wojciech Seliga](https://reader031.vdocument.in/reader031/viewer/2022013008/5e3af5ab1060d64d7023d879/html5/thumbnails/46.jpg)
Platforms
• Dimension - DB: MySQL, PostgreSQL, MS SQL, Oracle
• Dimension - OS: Linux, Windows
• Dimension - Java ver.: 1.5, 1.6, 1.7
• Dimension - CPU arch.: 32-bit, 64-bit
• Dimension - Deployment Mode: Standalone, Tomcat, Websphere, Weblogic
• Dimension - Browsers: IE 8+, FF, Chrome,
Coming
26
![Page 47: Wojciech Seliga wojciech.seliga@spartez.com, T: @wseligas3-eu-west-1.amazonaws.com/presentations2012/36_presentation.pdf · 5/16/12 How to survive automated tests hell Wojciech Seliga](https://reader031.vdocument.in/reader031/viewer/2022013008/5e3af5ab1060d64d7023d879/html5/thumbnails/47.jpg)
Platforms
• Dimension - DB: MySQL, PostgreSQL, MS SQL, Oracle
• Dimension - OS: Linux, Windows
• Dimension - Java ver.: 1.5, 1.6, 1.7
• Dimension - CPU arch.: 32-bit, 64-bit
• Dimension - Deployment Mode: Standalone, Tomcat, Websphere, Weblogic
• Dimension - Browsers: IE 8+, FF, Chrome,
Run Nightlyor Before Release
Coming
26
![Page 48: Wojciech Seliga wojciech.seliga@spartez.com, T: @wseligas3-eu-west-1.amazonaws.com/presentations2012/36_presentation.pdf · 5/16/12 How to survive automated tests hell Wojciech Seliga](https://reader031.vdocument.in/reader031/viewer/2022013008/5e3af5ab1060d64d7023d879/html5/thumbnails/48.jpg)
Triggering Builds
• On Commit (hooks, polling)
• Dependent Builds
• Nightly Builds
• Manual Builds
27
![Page 49: Wojciech Seliga wojciech.seliga@spartez.com, T: @wseligas3-eu-west-1.amazonaws.com/presentations2012/36_presentation.pdf · 5/16/12 How to survive automated tests hell Wojciech Seliga](https://reader031.vdocument.in/reader031/viewer/2022013008/5e3af5ab1060d64d7023d879/html5/thumbnails/49.jpg)
But...
28
![Page 50: Wojciech Seliga wojciech.seliga@spartez.com, T: @wseligas3-eu-west-1.amazonaws.com/presentations2012/36_presentation.pdf · 5/16/12 How to survive automated tests hell Wojciech Seliga](https://reader031.vdocument.in/reader031/viewer/2022013008/5e3af5ab1060d64d7023d879/html5/thumbnails/50.jpg)
Slow unit test29
![Page 51: Wojciech Seliga wojciech.seliga@spartez.com, T: @wseligas3-eu-west-1.amazonaws.com/presentations2012/36_presentation.pdf · 5/16/12 How to survive automated tests hell Wojciech Seliga](https://reader031.vdocument.in/reader031/viewer/2022013008/5e3af5ab1060d64d7023d879/html5/thumbnails/51.jpg)
Very slow functional tests30
![Page 52: Wojciech Seliga wojciech.seliga@spartez.com, T: @wseligas3-eu-west-1.amazonaws.com/presentations2012/36_presentation.pdf · 5/16/12 How to survive automated tests hell Wojciech Seliga](https://reader031.vdocument.in/reader031/viewer/2022013008/5e3af5ab1060d64d7023d879/html5/thumbnails/52.jpg)
Builds Wait in The Queue31
![Page 53: Wojciech Seliga wojciech.seliga@spartez.com, T: @wseligas3-eu-west-1.amazonaws.com/presentations2012/36_presentation.pdf · 5/16/12 How to survive automated tests hell Wojciech Seliga](https://reader031.vdocument.in/reader031/viewer/2022013008/5e3af5ab1060d64d7023d879/html5/thumbnails/53.jpg)
Builds Often Fail32
![Page 54: Wojciech Seliga wojciech.seliga@spartez.com, T: @wseligas3-eu-west-1.amazonaws.com/presentations2012/36_presentation.pdf · 5/16/12 How to survive automated tests hell Wojciech Seliga](https://reader031.vdocument.in/reader031/viewer/2022013008/5e3af5ab1060d64d7023d879/html5/thumbnails/54.jpg)
Too Often...33
![Page 55: Wojciech Seliga wojciech.seliga@spartez.com, T: @wseligas3-eu-west-1.amazonaws.com/presentations2012/36_presentation.pdf · 5/16/12 How to survive automated tests hell Wojciech Seliga](https://reader031.vdocument.in/reader031/viewer/2022013008/5e3af5ab1060d64d7023d879/html5/thumbnails/55.jpg)
It takes time to fix it...34
![Page 56: Wojciech Seliga wojciech.seliga@spartez.com, T: @wseligas3-eu-west-1.amazonaws.com/presentations2012/36_presentation.pdf · 5/16/12 How to survive automated tests hell Wojciech Seliga](https://reader031.vdocument.in/reader031/viewer/2022013008/5e3af5ab1060d64d7023d879/html5/thumbnails/56.jpg)
Sometimes very long35
![Page 57: Wojciech Seliga wojciech.seliga@spartez.com, T: @wseligas3-eu-west-1.amazonaws.com/presentations2012/36_presentation.pdf · 5/16/12 How to survive automated tests hell Wojciech Seliga](https://reader031.vdocument.in/reader031/viewer/2022013008/5e3af5ab1060d64d7023d879/html5/thumbnails/57.jpg)
You commit at 3 PM
36
![Page 58: Wojciech Seliga wojciech.seliga@spartez.com, T: @wseligas3-eu-west-1.amazonaws.com/presentations2012/36_presentation.pdf · 5/16/12 How to survive automated tests hell Wojciech Seliga](https://reader031.vdocument.in/reader031/viewer/2022013008/5e3af5ab1060d64d7023d879/html5/thumbnails/58.jpg)
You commit at 3 PM
You get “Unit Test Green” email at 4PM
36
![Page 59: Wojciech Seliga wojciech.seliga@spartez.com, T: @wseligas3-eu-west-1.amazonaws.com/presentations2012/36_presentation.pdf · 5/16/12 How to survive automated tests hell Wojciech Seliga](https://reader031.vdocument.in/reader031/viewer/2022013008/5e3af5ab1060d64d7023d879/html5/thumbnails/59.jpg)
You commit at 3 PM
You get “Unit Test Green” email at 4PM
You happily go home
36
![Page 60: Wojciech Seliga wojciech.seliga@spartez.com, T: @wseligas3-eu-west-1.amazonaws.com/presentations2012/36_presentation.pdf · 5/16/12 How to survive automated tests hell Wojciech Seliga](https://reader031.vdocument.in/reader031/viewer/2022013008/5e3af5ab1060d64d7023d879/html5/thumbnails/60.jpg)
You commit at 3 PM
You get “Unit Test Green” email at 4PM
You get flood of “Red Test X” emails at 4 - 9PM
You happily go home
36
![Page 61: Wojciech Seliga wojciech.seliga@spartez.com, T: @wseligas3-eu-west-1.amazonaws.com/presentations2012/36_presentation.pdf · 5/16/12 How to survive automated tests hell Wojciech Seliga](https://reader031.vdocument.in/reader031/viewer/2022013008/5e3af5ab1060d64d7023d879/html5/thumbnails/61.jpg)
You commit at 3 PM
You get “Unit Test Green” email at 4PM
You get flood of “Red Test X” emails at 4 - 9PM
Your colleagues on the other side of the globe
You happily go home
36
![Page 62: Wojciech Seliga wojciech.seliga@spartez.com, T: @wseligas3-eu-west-1.amazonaws.com/presentations2012/36_presentation.pdf · 5/16/12 How to survive automated tests hell Wojciech Seliga](https://reader031.vdocument.in/reader031/viewer/2022013008/5e3af5ab1060d64d7023d879/html5/thumbnails/62.jpg)
You commit at 3 PM
You get “Unit Test Green” email at 4PM
You get flood of “Red Test X” emails at 4 - 9PM
Your colleagues on the other side of the globe
You happily go home
You36
![Page 63: Wojciech Seliga wojciech.seliga@spartez.com, T: @wseligas3-eu-west-1.amazonaws.com/presentations2012/36_presentation.pdf · 5/16/12 How to survive automated tests hell Wojciech Seliga](https://reader031.vdocument.in/reader031/viewer/2022013008/5e3af5ab1060d64d7023d879/html5/thumbnails/63.jpg)
“Slow CI loop and non-deterministic tests are
strong inhibitor of change instead of the catalyst”
by W. Seliga37
![Page 64: Wojciech Seliga wojciech.seliga@spartez.com, T: @wseligas3-eu-west-1.amazonaws.com/presentations2012/36_presentation.pdf · 5/16/12 How to survive automated tests hell Wojciech Seliga](https://reader031.vdocument.in/reader031/viewer/2022013008/5e3af5ab1060d64d7023d879/html5/thumbnails/64.jpg)
“We probably spend more time dealing with the JIRA
test codebase than the production codebase”
38
![Page 65: Wojciech Seliga wojciech.seliga@spartez.com, T: @wseligas3-eu-west-1.amazonaws.com/presentations2012/36_presentation.pdf · 5/16/12 How to survive automated tests hell Wojciech Seliga](https://reader031.vdocument.in/reader031/viewer/2022013008/5e3af5ab1060d64d7023d879/html5/thumbnails/65.jpg)
0
25
50
75
100
Test Coverage
Effort Invested
Striving for Coverage
39
![Page 66: Wojciech Seliga wojciech.seliga@spartez.com, T: @wseligas3-eu-west-1.amazonaws.com/presentations2012/36_presentation.pdf · 5/16/12 How to survive automated tests hell Wojciech Seliga](https://reader031.vdocument.in/reader031/viewer/2022013008/5e3af5ab1060d64d7023d879/html5/thumbnails/66.jpg)
0%
25%
50%
75%
100%
Value
Size of automated tests
Strange? Relationship
40
![Page 67: Wojciech Seliga wojciech.seliga@spartez.com, T: @wseligas3-eu-west-1.amazonaws.com/presentations2012/36_presentation.pdf · 5/16/12 How to survive automated tests hell Wojciech Seliga](https://reader031.vdocument.in/reader031/viewer/2022013008/5e3af5ab1060d64d7023d879/html5/thumbnails/67.jpg)
Test Hell - The Outcomes
• Development slows down
• Devs are afraid of change
• Software difficult to release
• Significant amount of time spent on analysing test failures
• Morale goes down
41
![Page 68: Wojciech Seliga wojciech.seliga@spartez.com, T: @wseligas3-eu-west-1.amazonaws.com/presentations2012/36_presentation.pdf · 5/16/12 How to survive automated tests hell Wojciech Seliga](https://reader031.vdocument.in/reader031/viewer/2022013008/5e3af5ab1060d64d7023d879/html5/thumbnails/68.jpg)
FeedbackSpeed
`Quality
42
![Page 69: Wojciech Seliga wojciech.seliga@spartez.com, T: @wseligas3-eu-west-1.amazonaws.com/presentations2012/36_presentation.pdf · 5/16/12 How to survive automated tests hell Wojciech Seliga](https://reader031.vdocument.in/reader031/viewer/2022013008/5e3af5ab1060d64d7023d879/html5/thumbnails/69.jpg)
Feedback Loop Speed
• Tiniest change triggers test avalanche
• Lack of responsibility syndrome
• Devs do not run tests locally (speed)
• Before you get the results you are at home
43
![Page 70: Wojciech Seliga wojciech.seliga@spartez.com, T: @wseligas3-eu-west-1.amazonaws.com/presentations2012/36_presentation.pdf · 5/16/12 How to survive automated tests hell Wojciech Seliga](https://reader031.vdocument.in/reader031/viewer/2022013008/5e3af5ab1060d64d7023d879/html5/thumbnails/70.jpg)
Quality
• Non-deterministic tests (races, timeouts)
• Catching up with UI changes
• 1 red test hides new failures
• Ignoring always red tests in dangerous ...
44
![Page 71: Wojciech Seliga wojciech.seliga@spartez.com, T: @wseligas3-eu-west-1.amazonaws.com/presentations2012/36_presentation.pdf · 5/16/12 How to survive automated tests hell Wojciech Seliga](https://reader031.vdocument.in/reader031/viewer/2022013008/5e3af5ab1060d64d7023d879/html5/thumbnails/71.jpg)
Broken window theory45
![Page 72: Wojciech Seliga wojciech.seliga@spartez.com, T: @wseligas3-eu-west-1.amazonaws.com/presentations2012/36_presentation.pdf · 5/16/12 How to survive automated tests hell Wojciech Seliga](https://reader031.vdocument.in/reader031/viewer/2022013008/5e3af5ab1060d64d7023d879/html5/thumbnails/72.jpg)
Decisions which do not scale
46
![Page 73: Wojciech Seliga wojciech.seliga@spartez.com, T: @wseligas3-eu-west-1.amazonaws.com/presentations2012/36_presentation.pdf · 5/16/12 How to survive automated tests hell Wojciech Seliga](https://reader031.vdocument.in/reader031/viewer/2022013008/5e3af5ab1060d64d7023d879/html5/thumbnails/73.jpg)
Decisions which do not scale
• All unit tests in one maven module
46
![Page 74: Wojciech Seliga wojciech.seliga@spartez.com, T: @wseligas3-eu-west-1.amazonaws.com/presentations2012/36_presentation.pdf · 5/16/12 How to survive automated tests hell Wojciech Seliga](https://reader031.vdocument.in/reader031/viewer/2022013008/5e3af5ab1060d64d7023d879/html5/thumbnails/74.jpg)
Decisions which do not scale
• All unit tests in one maven module
• All functional tests in one maven module
46
![Page 75: Wojciech Seliga wojciech.seliga@spartez.com, T: @wseligas3-eu-west-1.amazonaws.com/presentations2012/36_presentation.pdf · 5/16/12 How to survive automated tests hell Wojciech Seliga](https://reader031.vdocument.in/reader031/viewer/2022013008/5e3af5ab1060d64d7023d879/html5/thumbnails/75.jpg)
Decisions which do not scale
• All unit tests in one maven module
• All functional tests in one maven module
• All Selenium and web-driver tests in one module
46
![Page 76: Wojciech Seliga wojciech.seliga@spartez.com, T: @wseligas3-eu-west-1.amazonaws.com/presentations2012/36_presentation.pdf · 5/16/12 How to survive automated tests hell Wojciech Seliga](https://reader031.vdocument.in/reader031/viewer/2022013008/5e3af5ab1060d64d7023d879/html5/thumbnails/76.jpg)
Decisions which do not scale
• All unit tests in one maven module
• All functional tests in one maven module
• All Selenium and web-driver tests in one module
• Every commit triggers rebuild and re-test of everything
46
![Page 77: Wojciech Seliga wojciech.seliga@spartez.com, T: @wseligas3-eu-west-1.amazonaws.com/presentations2012/36_presentation.pdf · 5/16/12 How to survive automated tests hell Wojciech Seliga](https://reader031.vdocument.in/reader031/viewer/2022013008/5e3af5ab1060d64d7023d879/html5/thumbnails/77.jpg)
Decisions which do not scale
• All unit tests in one maven module
• All functional tests in one maven module
• All Selenium and web-driver tests in one module
• Every commit triggers rebuild and re-test of everything
• Monolithic test framework / utils
46
![Page 78: Wojciech Seliga wojciech.seliga@spartez.com, T: @wseligas3-eu-west-1.amazonaws.com/presentations2012/36_presentation.pdf · 5/16/12 How to survive automated tests hell Wojciech Seliga](https://reader031.vdocument.in/reader031/viewer/2022013008/5e3af5ab1060d64d7023d879/html5/thumbnails/78.jpg)
Decisions which do not scale
• All unit tests in one maven module
• All functional tests in one maven module
• All Selenium and web-driver tests in one module
• Every commit triggers rebuild and re-test of everything
• Monolithic test framework / utils
• Opaque fixtures
46
![Page 79: Wojciech Seliga wojciech.seliga@spartez.com, T: @wseligas3-eu-west-1.amazonaws.com/presentations2012/36_presentation.pdf · 5/16/12 How to survive automated tests hell Wojciech Seliga](https://reader031.vdocument.in/reader031/viewer/2022013008/5e3af5ab1060d64d7023d879/html5/thumbnails/79.jpg)
Strategies
47
![Page 80: Wojciech Seliga wojciech.seliga@spartez.com, T: @wseligas3-eu-west-1.amazonaws.com/presentations2012/36_presentation.pdf · 5/16/12 How to survive automated tests hell Wojciech Seliga](https://reader031.vdocument.in/reader031/viewer/2022013008/5e3af5ab1060d64d7023d879/html5/thumbnails/80.jpg)
Test Quality
48
![Page 81: Wojciech Seliga wojciech.seliga@spartez.com, T: @wseligas3-eu-west-1.amazonaws.com/presentations2012/36_presentation.pdf · 5/16/12 How to survive automated tests hell Wojciech Seliga](https://reader031.vdocument.in/reader031/viewer/2022013008/5e3af5ab1060d64d7023d879/html5/thumbnails/81.jpg)
Catching up with UI changes
Problem:
49
![Page 82: Wojciech Seliga wojciech.seliga@spartez.com, T: @wseligas3-eu-west-1.amazonaws.com/presentations2012/36_presentation.pdf · 5/16/12 How to survive automated tests hell Wojciech Seliga](https://reader031.vdocument.in/reader031/viewer/2022013008/5e3af5ab1060d64d7023d879/html5/thumbnails/82.jpg)
Catching up with UI changes
Problem:
Solution:
49
![Page 83: Wojciech Seliga wojciech.seliga@spartez.com, T: @wseligas3-eu-west-1.amazonaws.com/presentations2012/36_presentation.pdf · 5/16/12 How to survive automated tests hell Wojciech Seliga](https://reader031.vdocument.in/reader031/viewer/2022013008/5e3af5ab1060d64d7023d879/html5/thumbnails/83.jpg)
Catching up with UI changes
Page Objects Pattern
Problem:
Solution:
49
![Page 84: Wojciech Seliga wojciech.seliga@spartez.com, T: @wseligas3-eu-west-1.amazonaws.com/presentations2012/36_presentation.pdf · 5/16/12 How to survive automated tests hell Wojciech Seliga](https://reader031.vdocument.in/reader031/viewer/2022013008/5e3af5ab1060d64d7023d879/html5/thumbnails/84.jpg)
Page Objects Pattern• Page Objects model UI elements (pages,
components, dialogs, areas) your tests interact with
• Page Objects shield tests from changing internal structure of the page
• Page Objects generally do not make assertions about data. The can assert the state.
• Designed for chaining50
![Page 85: Wojciech Seliga wojciech.seliga@spartez.com, T: @wseligas3-eu-west-1.amazonaws.com/presentations2012/36_presentation.pdf · 5/16/12 How to survive automated tests hell Wojciech Seliga](https://reader031.vdocument.in/reader031/viewer/2022013008/5e3af5ab1060d64d7023d879/html5/thumbnails/85.jpg)
Page Objects Examplepublic class AddUserPage extends AbstractJiraPage{
private static final String URI = "/secure/admin/user/AddUser!default.jspa";
@ElementBy(name = "username") private PageElement username;
@ElementBy(name = "password") private PageElement password;
@ElementBy(name = "confirm") private PageElement passwordConfirmation;
@ElementBy(name = "fullname") private PageElement fullName;
@ElementBy(name = "email") private PageElement email;
@ElementBy(name = "sendemail") private PageElement sendEmail;
@ElementBy(id = "user-create-submit") private PageElement submit;
@ElementBy (id = "user-create-cancel") private PageElement cancelButton;
@Override public String getUrl() { return URI; }
...
@Override public TimedCondition isAt() { return and(username.timed().isPresent(), password.timed().isPresent(), fullName.timed().isPresent()); }
public AddUserPage addUser(final String username, final String password, final String fullName, final String email, final boolean receiveEmail)
{ this.username.type(username); this.password.type(password); this.passwordConfirmation.type(password); this.fullName.type(fullName); this.email.type(email); if(receiveEmail) { this.sendEmail.select(); } return this; }
public ViewUserPage createUser() { return createUser(ViewUserPage.class); }
public <T extends Page> T createUser(Class<T> nextPage, Object...args) { submit.click(); return pageBinder.bind(nextPage, args); }
51
![Page 86: Wojciech Seliga wojciech.seliga@spartez.com, T: @wseligas3-eu-west-1.amazonaws.com/presentations2012/36_presentation.pdf · 5/16/12 How to survive automated tests hell Wojciech Seliga](https://reader031.vdocument.in/reader031/viewer/2022013008/5e3af5ab1060d64d7023d879/html5/thumbnails/86.jpg)
Using Page Objects @Test public void testServerError() { jira.gotoLoginPage().loginAsSysAdmin(AddUserPage.class) .addUser("username", "mypassword", "My Name",
"[email protected]", false) .createUser();
// assertions here }
52
![Page 87: Wojciech Seliga wojciech.seliga@spartez.com, T: @wseligas3-eu-west-1.amazonaws.com/presentations2012/36_presentation.pdf · 5/16/12 How to survive automated tests hell Wojciech Seliga](https://reader031.vdocument.in/reader031/viewer/2022013008/5e3af5ab1060d64d7023d879/html5/thumbnails/87.jpg)
Using Page Objects @Test public void testImportSampleProject() { final PivotalImporterSetupPage setupPage = getSetupPage(); assertEquals("1. Connect", setupPage.getActiveTabText());
final PivotalProjectsMappingsPage projectMappingPage = setupPage.next(); assertEquals("2. Project Mapping", setupPage.getActiveTabText()); assertTrue("Expecting all project to be selected by default", projectMappingPage.areAllProjectsSelected()); projectMappingPage.setImportAllProjects(false); projectMappingPage.setProjectImported(sampleProject, true); projectMappingPage.createProject(sampleProject, sampleProject, "SAMPLE");
final ImporterFinishedPage importerLogsPage = projectMappingPage.beginImport().waitUntilFinished(); assertTrue(importerLogsPage.isSuccess()); assertEquals(0, importerLogsPage.getGlobalErrors().size()); assertEquals("1", importerLogsPage.getProjectsImported()); // more assertions here }
53
![Page 88: Wojciech Seliga wojciech.seliga@spartez.com, T: @wseligas3-eu-west-1.amazonaws.com/presentations2012/36_presentation.pdf · 5/16/12 How to survive automated tests hell Wojciech Seliga](https://reader031.vdocument.in/reader031/viewer/2022013008/5e3af5ab1060d64d7023d879/html5/thumbnails/88.jpg)
More on Page Objects
• Design for reusability
• Design for sharing - libraries of Page Objects
• Good support by WebDriver/Selenium 2
• Atlassian Selenium 2.0
54
![Page 89: Wojciech Seliga wojciech.seliga@spartez.com, T: @wseligas3-eu-west-1.amazonaws.com/presentations2012/36_presentation.pdf · 5/16/12 How to survive automated tests hell Wojciech Seliga](https://reader031.vdocument.in/reader031/viewer/2022013008/5e3af5ab1060d64d7023d879/html5/thumbnails/89.jpg)
Page Objects Caveat
The easier is to write functional tests, the more tests will be written...
The more stuff you will need to maintain,the more time the execution of your tests will take...
W. Seliga, PO. 1-4
55
![Page 90: Wojciech Seliga wojciech.seliga@spartez.com, T: @wseligas3-eu-west-1.amazonaws.com/presentations2012/36_presentation.pdf · 5/16/12 How to survive automated tests hell Wojciech Seliga](https://reader031.vdocument.in/reader031/viewer/2022013008/5e3af5ab1060d64d7023d879/html5/thumbnails/90.jpg)
Opaque Test FixturesProblem:
56
![Page 91: Wojciech Seliga wojciech.seliga@spartez.com, T: @wseligas3-eu-west-1.amazonaws.com/presentations2012/36_presentation.pdf · 5/16/12 How to survive automated tests hell Wojciech Seliga](https://reader031.vdocument.in/reader031/viewer/2022013008/5e3af5ab1060d64d7023d879/html5/thumbnails/91.jpg)
Opaque Test FixturesProblem:
Solution:
56
![Page 92: Wojciech Seliga wojciech.seliga@spartez.com, T: @wseligas3-eu-west-1.amazonaws.com/presentations2012/36_presentation.pdf · 5/16/12 How to survive automated tests hell Wojciech Seliga](https://reader031.vdocument.in/reader031/viewer/2022013008/5e3af5ab1060d64d7023d879/html5/thumbnails/92.jpg)
Opaque Test Fixtures
REST-based Set-up,Backdoors
Problem:
Solution:
56
![Page 93: Wojciech Seliga wojciech.seliga@spartez.com, T: @wseligas3-eu-west-1.amazonaws.com/presentations2012/36_presentation.pdf · 5/16/12 How to survive automated tests hell Wojciech Seliga](https://reader031.vdocument.in/reader031/viewer/2022013008/5e3af5ab1060d64d7023d879/html5/thumbnails/93.jpg)
REST-based Setup
57
![Page 94: Wojciech Seliga wojciech.seliga@spartez.com, T: @wseligas3-eu-west-1.amazonaws.com/presentations2012/36_presentation.pdf · 5/16/12 How to survive automated tests hell Wojciech Seliga](https://reader031.vdocument.in/reader031/viewer/2022013008/5e3af5ab1060d64d7023d879/html5/thumbnails/94.jpg)
REST-based Setup @Before public void setUpTest() { restore("some-big-xml-file-with-everything-needed-inside.xml"); }
57
![Page 95: Wojciech Seliga wojciech.seliga@spartez.com, T: @wseligas3-eu-west-1.amazonaws.com/presentations2012/36_presentation.pdf · 5/16/12 How to survive automated tests hell Wojciech Seliga](https://reader031.vdocument.in/reader031/viewer/2022013008/5e3af5ab1060d64d7023d879/html5/thumbnails/95.jpg)
REST-based Setup @Before public void setUpTest() { restore("some-big-xml-file-with-everything-needed-inside.xml"); }
VS
57
![Page 96: Wojciech Seliga wojciech.seliga@spartez.com, T: @wseligas3-eu-west-1.amazonaws.com/presentations2012/36_presentation.pdf · 5/16/12 How to survive automated tests hell Wojciech Seliga](https://reader031.vdocument.in/reader031/viewer/2022013008/5e3af5ab1060d64d7023d879/html5/thumbnails/96.jpg)
REST-based Setup @Before public void setUpTest() { restore("some-big-xml-file-with-everything-needed-inside.xml"); }
@Before public void setUpTest() { restClient.restoreEmptyInstance(); restClient.createProject(/* project params */); restClient.createUser(/* user params */); restClient.createUser(/* user params */); restClient.createSomethingElse(/* ... */); }
VS
57
![Page 97: Wojciech Seliga wojciech.seliga@spartez.com, T: @wseligas3-eu-west-1.amazonaws.com/presentations2012/36_presentation.pdf · 5/16/12 How to survive automated tests hell Wojciech Seliga](https://reader031.vdocument.in/reader031/viewer/2022013008/5e3af5ab1060d64d7023d879/html5/thumbnails/97.jpg)
Flakey TestsProblem:
58
![Page 98: Wojciech Seliga wojciech.seliga@spartez.com, T: @wseligas3-eu-west-1.amazonaws.com/presentations2012/36_presentation.pdf · 5/16/12 How to survive automated tests hell Wojciech Seliga](https://reader031.vdocument.in/reader031/viewer/2022013008/5e3af5ab1060d64d7023d879/html5/thumbnails/98.jpg)
Flakey TestsProblem:
Solution:
58
![Page 99: Wojciech Seliga wojciech.seliga@spartez.com, T: @wseligas3-eu-west-1.amazonaws.com/presentations2012/36_presentation.pdf · 5/16/12 How to survive automated tests hell Wojciech Seliga](https://reader031.vdocument.in/reader031/viewer/2022013008/5e3af5ab1060d64d7023d879/html5/thumbnails/99.jpg)
Flakey Tests
Quarantine
Problem:
Solution:
58
![Page 100: Wojciech Seliga wojciech.seliga@spartez.com, T: @wseligas3-eu-west-1.amazonaws.com/presentations2012/36_presentation.pdf · 5/16/12 How to survive automated tests hell Wojciech Seliga](https://reader031.vdocument.in/reader031/viewer/2022013008/5e3af5ab1060d64d7023d879/html5/thumbnails/100.jpg)
Flakey Tests
Quarantine
Problem:
Solution:
Fix58
![Page 101: Wojciech Seliga wojciech.seliga@spartez.com, T: @wseligas3-eu-west-1.amazonaws.com/presentations2012/36_presentation.pdf · 5/16/12 How to survive automated tests hell Wojciech Seliga](https://reader031.vdocument.in/reader031/viewer/2022013008/5e3af5ab1060d64d7023d879/html5/thumbnails/101.jpg)
Flakey Tests
Quarantine
Problem:
Solution:
Fix Eradicate58
![Page 102: Wojciech Seliga wojciech.seliga@spartez.com, T: @wseligas3-eu-west-1.amazonaws.com/presentations2012/36_presentation.pdf · 5/16/12 How to survive automated tests hell Wojciech Seliga](https://reader031.vdocument.in/reader031/viewer/2022013008/5e3af5ab1060d64d7023d879/html5/thumbnails/102.jpg)
Quarantine• @Ignore, @Category
• Quarantine on CI server
• Recover or Die - limiting quarantine
59
![Page 103: Wojciech Seliga wojciech.seliga@spartez.com, T: @wseligas3-eu-west-1.amazonaws.com/presentations2012/36_presentation.pdf · 5/16/12 How to survive automated tests hell Wojciech Seliga](https://reader031.vdocument.in/reader031/viewer/2022013008/5e3af5ab1060d64d7023d879/html5/thumbnails/103.jpg)
Quarantine• @Ignore, @Category
• Quarantine on CI server
• Recover or Die - limiting quarantine
59
![Page 104: Wojciech Seliga wojciech.seliga@spartez.com, T: @wseligas3-eu-west-1.amazonaws.com/presentations2012/36_presentation.pdf · 5/16/12 How to survive automated tests hell Wojciech Seliga](https://reader031.vdocument.in/reader031/viewer/2022013008/5e3af5ab1060d64d7023d879/html5/thumbnails/104.jpg)
Quarantine• @Ignore, @Category
• Quarantine on CI server
• Recover or Die - limiting quarantine
59
![Page 105: Wojciech Seliga wojciech.seliga@spartez.com, T: @wseligas3-eu-west-1.amazonaws.com/presentations2012/36_presentation.pdf · 5/16/12 How to survive automated tests hell Wojciech Seliga](https://reader031.vdocument.in/reader031/viewer/2022013008/5e3af5ab1060d64d7023d879/html5/thumbnails/105.jpg)
Heal or Kill
60
![Page 106: Wojciech Seliga wojciech.seliga@spartez.com, T: @wseligas3-eu-west-1.amazonaws.com/presentations2012/36_presentation.pdf · 5/16/12 How to survive automated tests hell Wojciech Seliga](https://reader031.vdocument.in/reader031/viewer/2022013008/5e3af5ab1060d64d7023d879/html5/thumbnails/106.jpg)
Heal or Kill
60
Quickly!
![Page 107: Wojciech Seliga wojciech.seliga@spartez.com, T: @wseligas3-eu-west-1.amazonaws.com/presentations2012/36_presentation.pdf · 5/16/12 How to survive automated tests hell Wojciech Seliga](https://reader031.vdocument.in/reader031/viewer/2022013008/5e3af5ab1060d64d7023d879/html5/thumbnails/107.jpg)
Fixing Flakey TestsProblem:
61
![Page 108: Wojciech Seliga wojciech.seliga@spartez.com, T: @wseligas3-eu-west-1.amazonaws.com/presentations2012/36_presentation.pdf · 5/16/12 How to survive automated tests hell Wojciech Seliga](https://reader031.vdocument.in/reader031/viewer/2022013008/5e3af5ab1060d64d7023d879/html5/thumbnails/108.jpg)
Fixing Flakey TestsProblem:
Solution:
61
![Page 109: Wojciech Seliga wojciech.seliga@spartez.com, T: @wseligas3-eu-west-1.amazonaws.com/presentations2012/36_presentation.pdf · 5/16/12 How to survive automated tests hell Wojciech Seliga](https://reader031.vdocument.in/reader031/viewer/2022013008/5e3af5ab1060d64d7023d879/html5/thumbnails/109.jpg)
Fixing Flakey Tests
Timed Conditions
Problem:
Solution:
61
![Page 110: Wojciech Seliga wojciech.seliga@spartez.com, T: @wseligas3-eu-west-1.amazonaws.com/presentations2012/36_presentation.pdf · 5/16/12 How to survive automated tests hell Wojciech Seliga](https://reader031.vdocument.in/reader031/viewer/2022013008/5e3af5ab1060d64d7023d879/html5/thumbnails/110.jpg)
Fixing Flakey Tests
Timed Conditions
Problem:
Solution:
Test-friendly Markup61
![Page 111: Wojciech Seliga wojciech.seliga@spartez.com, T: @wseligas3-eu-west-1.amazonaws.com/presentations2012/36_presentation.pdf · 5/16/12 How to survive automated tests hell Wojciech Seliga](https://reader031.vdocument.in/reader031/viewer/2022013008/5e3af5ab1060d64d7023d879/html5/thumbnails/111.jpg)
Fixing Flakey Tests
Timed Conditions
Problem:
Solution:
Mock Unreliable DepsTest-friendly Markup
61
![Page 112: Wojciech Seliga wojciech.seliga@spartez.com, T: @wseligas3-eu-west-1.amazonaws.com/presentations2012/36_presentation.pdf · 5/16/12 How to survive automated tests hell Wojciech Seliga](https://reader031.vdocument.in/reader031/viewer/2022013008/5e3af5ab1060d64d7023d879/html5/thumbnails/112.jpg)
Timed Conditions
62
![Page 113: Wojciech Seliga wojciech.seliga@spartez.com, T: @wseligas3-eu-west-1.amazonaws.com/presentations2012/36_presentation.pdf · 5/16/12 How to survive automated tests hell Wojciech Seliga](https://reader031.vdocument.in/reader031/viewer/2022013008/5e3af5ab1060d64d7023d879/html5/thumbnails/113.jpg)
Test-friendly Markup
• Do not save on IDs
• Do not save on CSS classes
• XPath is fragile
• XPath is expensive
• XPath is not readable
• i18N
63
![Page 114: Wojciech Seliga wojciech.seliga@spartez.com, T: @wseligas3-eu-west-1.amazonaws.com/presentations2012/36_presentation.pdf · 5/16/12 How to survive automated tests hell Wojciech Seliga](https://reader031.vdocument.in/reader031/viewer/2022013008/5e3af5ab1060d64d7023d879/html5/thumbnails/114.jpg)
Mock Unreliable Dependencies
64
![Page 115: Wojciech Seliga wojciech.seliga@spartez.com, T: @wseligas3-eu-west-1.amazonaws.com/presentations2012/36_presentation.pdf · 5/16/12 How to survive automated tests hell Wojciech Seliga](https://reader031.vdocument.in/reader031/viewer/2022013008/5e3af5ab1060d64d7023d879/html5/thumbnails/115.jpg)
Speed
65
![Page 116: Wojciech Seliga wojciech.seliga@spartez.com, T: @wseligas3-eu-west-1.amazonaws.com/presentations2012/36_presentation.pdf · 5/16/12 How to survive automated tests hell Wojciech Seliga](https://reader031.vdocument.in/reader031/viewer/2022013008/5e3af5ab1060d64d7023d879/html5/thumbnails/116.jpg)
SpeedAiming at 10 seconds build
65
![Page 117: Wojciech Seliga wojciech.seliga@spartez.com, T: @wseligas3-eu-west-1.amazonaws.com/presentations2012/36_presentation.pdf · 5/16/12 How to survive automated tests hell Wojciech Seliga](https://reader031.vdocument.in/reader031/viewer/2022013008/5e3af5ab1060d64d7023d879/html5/thumbnails/117.jpg)
Splitting CodebaseThe easiest and most effective
improvement
66
![Page 118: Wojciech Seliga wojciech.seliga@spartez.com, T: @wseligas3-eu-west-1.amazonaws.com/presentations2012/36_presentation.pdf · 5/16/12 How to survive automated tests hell Wojciech Seliga](https://reader031.vdocument.in/reader031/viewer/2022013008/5e3af5ab1060d64d7023d879/html5/thumbnails/118.jpg)
Splitting Codebase
• Tests closer to tested code
• Less to test
• Testing less frequently
• Increased team responsibility
• Restructuring CI hierarchy - more complicated picture
67
![Page 119: Wojciech Seliga wojciech.seliga@spartez.com, T: @wseligas3-eu-west-1.amazonaws.com/presentations2012/36_presentation.pdf · 5/16/12 How to survive automated tests hell Wojciech Seliga](https://reader031.vdocument.in/reader031/viewer/2022013008/5e3af5ab1060d64d7023d879/html5/thumbnails/119.jpg)
Less to Test
68
![Page 120: Wojciech Seliga wojciech.seliga@spartez.com, T: @wseligas3-eu-west-1.amazonaws.com/presentations2012/36_presentation.pdf · 5/16/12 How to survive automated tests hell Wojciech Seliga](https://reader031.vdocument.in/reader031/viewer/2022013008/5e3af5ab1060d64d7023d879/html5/thumbnails/120.jpg)
Less to TestCommit
68
![Page 121: Wojciech Seliga wojciech.seliga@spartez.com, T: @wseligas3-eu-west-1.amazonaws.com/presentations2012/36_presentation.pdf · 5/16/12 How to survive automated tests hell Wojciech Seliga](https://reader031.vdocument.in/reader031/viewer/2022013008/5e3af5ab1060d64d7023d879/html5/thumbnails/121.jpg)
Less to TestCommit
68
![Page 122: Wojciech Seliga wojciech.seliga@spartez.com, T: @wseligas3-eu-west-1.amazonaws.com/presentations2012/36_presentation.pdf · 5/16/12 How to survive automated tests hell Wojciech Seliga](https://reader031.vdocument.in/reader031/viewer/2022013008/5e3af5ab1060d64d7023d879/html5/thumbnails/122.jpg)
Less to TestCommit
68
![Page 123: Wojciech Seliga wojciech.seliga@spartez.com, T: @wseligas3-eu-west-1.amazonaws.com/presentations2012/36_presentation.pdf · 5/16/12 How to survive automated tests hell Wojciech Seliga](https://reader031.vdocument.in/reader031/viewer/2022013008/5e3af5ab1060d64d7023d879/html5/thumbnails/123.jpg)
Less to TestCommit
68
![Page 124: Wojciech Seliga wojciech.seliga@spartez.com, T: @wseligas3-eu-west-1.amazonaws.com/presentations2012/36_presentation.pdf · 5/16/12 How to survive automated tests hell Wojciech Seliga](https://reader031.vdocument.in/reader031/viewer/2022013008/5e3af5ab1060d64d7023d879/html5/thumbnails/124.jpg)
Less to Test
68
![Page 125: Wojciech Seliga wojciech.seliga@spartez.com, T: @wseligas3-eu-west-1.amazonaws.com/presentations2012/36_presentation.pdf · 5/16/12 How to survive automated tests hell Wojciech Seliga](https://reader031.vdocument.in/reader031/viewer/2022013008/5e3af5ab1060d64d7023d879/html5/thumbnails/125.jpg)
Less to Test
Most of commits happen here68
![Page 126: Wojciech Seliga wojciech.seliga@spartez.com, T: @wseligas3-eu-west-1.amazonaws.com/presentations2012/36_presentation.pdf · 5/16/12 How to survive automated tests hell Wojciech Seliga](https://reader031.vdocument.in/reader031/viewer/2022013008/5e3af5ab1060d64d7023d879/html5/thumbnails/126.jpg)
Speed vs Control Workspace Dilemma
• Incubation
• Maturity
• Custom workspaces
69
![Page 127: Wojciech Seliga wojciech.seliga@spartez.com, T: @wseligas3-eu-west-1.amazonaws.com/presentations2012/36_presentation.pdf · 5/16/12 How to survive automated tests hell Wojciech Seliga](https://reader031.vdocument.in/reader031/viewer/2022013008/5e3af5ab1060d64d7023d879/html5/thumbnails/127.jpg)
Test Execution Time
70
![Page 128: Wojciech Seliga wojciech.seliga@spartez.com, T: @wseligas3-eu-west-1.amazonaws.com/presentations2012/36_presentation.pdf · 5/16/12 How to survive automated tests hell Wojciech Seliga](https://reader031.vdocument.in/reader031/viewer/2022013008/5e3af5ab1060d64d7023d879/html5/thumbnails/128.jpg)
Execution Time:Test Level
71
![Page 129: Wojciech Seliga wojciech.seliga@spartez.com, T: @wseligas3-eu-west-1.amazonaws.com/presentations2012/36_presentation.pdf · 5/16/12 How to survive automated tests hell Wojciech Seliga](https://reader031.vdocument.in/reader031/viewer/2022013008/5e3af5ab1060d64d7023d879/html5/thumbnails/129.jpg)
Execution Time:Test Level
Unit Tests
71
![Page 130: Wojciech Seliga wojciech.seliga@spartez.com, T: @wseligas3-eu-west-1.amazonaws.com/presentations2012/36_presentation.pdf · 5/16/12 How to survive automated tests hell Wojciech Seliga](https://reader031.vdocument.in/reader031/viewer/2022013008/5e3af5ab1060d64d7023d879/html5/thumbnails/130.jpg)
Execution Time:Test Level
Unit Tests
REST API Tests
71
![Page 131: Wojciech Seliga wojciech.seliga@spartez.com, T: @wseligas3-eu-west-1.amazonaws.com/presentations2012/36_presentation.pdf · 5/16/12 How to survive automated tests hell Wojciech Seliga](https://reader031.vdocument.in/reader031/viewer/2022013008/5e3af5ab1060d64d7023d879/html5/thumbnails/131.jpg)
Execution Time:Test Level
Unit Tests
REST API Tests
JWebUnit/HTMLUnit Tests
71
![Page 132: Wojciech Seliga wojciech.seliga@spartez.com, T: @wseligas3-eu-west-1.amazonaws.com/presentations2012/36_presentation.pdf · 5/16/12 How to survive automated tests hell Wojciech Seliga](https://reader031.vdocument.in/reader031/viewer/2022013008/5e3af5ab1060d64d7023d879/html5/thumbnails/132.jpg)
Execution Time:Test Level
Unit Tests
REST API Tests
JWebUnit/HTMLUnit Tests
Selenium/WebDriver Tests
71
![Page 133: Wojciech Seliga wojciech.seliga@spartez.com, T: @wseligas3-eu-west-1.amazonaws.com/presentations2012/36_presentation.pdf · 5/16/12 How to survive automated tests hell Wojciech Seliga](https://reader031.vdocument.in/reader031/viewer/2022013008/5e3af5ab1060d64d7023d879/html5/thumbnails/133.jpg)
Execution Time:Test Level
Unit Tests
REST API Tests
JWebUnit/HTMLUnit Tests
Selenium/WebDriver Tests
71
![Page 134: Wojciech Seliga wojciech.seliga@spartez.com, T: @wseligas3-eu-west-1.amazonaws.com/presentations2012/36_presentation.pdf · 5/16/12 How to survive automated tests hell Wojciech Seliga](https://reader031.vdocument.in/reader031/viewer/2022013008/5e3af5ab1060d64d7023d879/html5/thumbnails/134.jpg)
Execution Time:Test Level
Unit Tests
REST API Tests
JWebUnit/HTMLUnit Tests
Selenium/WebDriver Tests
Speed
71
![Page 135: Wojciech Seliga wojciech.seliga@spartez.com, T: @wseligas3-eu-west-1.amazonaws.com/presentations2012/36_presentation.pdf · 5/16/12 How to survive automated tests hell Wojciech Seliga](https://reader031.vdocument.in/reader031/viewer/2022013008/5e3af5ab1060d64d7023d879/html5/thumbnails/135.jpg)
Execution Time:Test Level
Unit Tests
REST API Tests
JWebUnit/HTMLUnit Tests
Selenium/WebDriver Tests
Speed
71
![Page 136: Wojciech Seliga wojciech.seliga@spartez.com, T: @wseligas3-eu-west-1.amazonaws.com/presentations2012/36_presentation.pdf · 5/16/12 How to survive automated tests hell Wojciech Seliga](https://reader031.vdocument.in/reader031/viewer/2022013008/5e3af5ab1060d64d7023d879/html5/thumbnails/136.jpg)
Execution Time:Test Level
Unit Tests
REST API Tests
JWebUnit/HTMLUnit Tests
Selenium/WebDriver Tests
Speed Confidence
71
![Page 137: Wojciech Seliga wojciech.seliga@spartez.com, T: @wseligas3-eu-west-1.amazonaws.com/presentations2012/36_presentation.pdf · 5/16/12 How to survive automated tests hell Wojciech Seliga](https://reader031.vdocument.in/reader031/viewer/2022013008/5e3af5ab1060d64d7023d879/html5/thumbnails/137.jpg)
Test Pyramid
Unit Tests
REST / HTML Tests
Selenium
72
![Page 138: Wojciech Seliga wojciech.seliga@spartez.com, T: @wseligas3-eu-west-1.amazonaws.com/presentations2012/36_presentation.pdf · 5/16/12 How to survive automated tests hell Wojciech Seliga](https://reader031.vdocument.in/reader031/viewer/2022013008/5e3af5ab1060d64d7023d879/html5/thumbnails/138.jpg)
Test Pyramid
Unit Tests
REST / HTML Tests
Selenium
72
![Page 139: Wojciech Seliga wojciech.seliga@spartez.com, T: @wseligas3-eu-west-1.amazonaws.com/presentations2012/36_presentation.pdf · 5/16/12 How to survive automated tests hell Wojciech Seliga](https://reader031.vdocument.in/reader031/viewer/2022013008/5e3af5ab1060d64d7023d879/html5/thumbnails/139.jpg)
Execution time - Cont.
• Batching
• Several tests per single set-up (violation of test isolation)
• REST-based setups and assertions
• Remove / merge overlapping tests
73
![Page 140: Wojciech Seliga wojciech.seliga@spartez.com, T: @wseligas3-eu-west-1.amazonaws.com/presentations2012/36_presentation.pdf · 5/16/12 How to survive automated tests hell Wojciech Seliga](https://reader031.vdocument.in/reader031/viewer/2022013008/5e3af5ab1060d64d7023d879/html5/thumbnails/140.jpg)
Execution time - Cont.
• IDs over CSS/JQuery Selectors over XPath
• JUnit tests running in the container
• In-process testing
• In-memory DBs
• Mocking web servers
• Reducing framework initialization time
• Test Optimization (Clover)
74
![Page 141: Wojciech Seliga wojciech.seliga@spartez.com, T: @wseligas3-eu-west-1.amazonaws.com/presentations2012/36_presentation.pdf · 5/16/12 How to survive automated tests hell Wojciech Seliga](https://reader031.vdocument.in/reader031/viewer/2022013008/5e3af5ab1060d64d7023d879/html5/thumbnails/141.jpg)
Waiting time
• More build agents
• Shorter and smaller tests
• No sleep()
• Avoiding long fixture setup (hot container, fragmented setup)
75
![Page 142: Wojciech Seliga wojciech.seliga@spartez.com, T: @wseligas3-eu-west-1.amazonaws.com/presentations2012/36_presentation.pdf · 5/16/12 How to survive automated tests hell Wojciech Seliga](https://reader031.vdocument.in/reader031/viewer/2022013008/5e3af5ab1060d64d7023d879/html5/thumbnails/142.jpg)
Preparation Time
• SCM performance
• Container set-up
• Compilation time (GWT...)
• Maven...
• Artifacts passing
76
![Page 143: Wojciech Seliga wojciech.seliga@spartez.com, T: @wseligas3-eu-west-1.amazonaws.com/presentations2012/36_presentation.pdf · 5/16/12 How to survive automated tests hell Wojciech Seliga](https://reader031.vdocument.in/reader031/viewer/2022013008/5e3af5ab1060d64d7023d879/html5/thumbnails/143.jpg)
So how about the goals?
Is 10 seconds build realistic?
77
![Page 144: Wojciech Seliga wojciech.seliga@spartez.com, T: @wseligas3-eu-west-1.amazonaws.com/presentations2012/36_presentation.pdf · 5/16/12 How to survive automated tests hell Wojciech Seliga](https://reader031.vdocument.in/reader031/viewer/2022013008/5e3af5ab1060d64d7023d879/html5/thumbnails/144.jpg)
Realistic Goals (for us*)
78
![Page 145: Wojciech Seliga wojciech.seliga@spartez.com, T: @wseligas3-eu-west-1.amazonaws.com/presentations2012/36_presentation.pdf · 5/16/12 How to survive automated tests hell Wojciech Seliga](https://reader031.vdocument.in/reader031/viewer/2022013008/5e3af5ab1060d64d7023d879/html5/thumbnails/145.jpg)
Realistic Goals (for us*)
*My current personal dreams78
![Page 146: Wojciech Seliga wojciech.seliga@spartez.com, T: @wseligas3-eu-west-1.amazonaws.com/presentations2012/36_presentation.pdf · 5/16/12 How to survive automated tests hell Wojciech Seliga](https://reader031.vdocument.in/reader031/viewer/2022013008/5e3af5ab1060d64d7023d879/html5/thumbnails/146.jpg)
Realistic Goals (for us*)
Time Type
2 min for unit tests for 95% of the commits
5 min for base smoke functional tests for 95% of the commits
15 min for ALL tests for 95% of the commits on selected platform
30 min for ALL tests for ALL commits
*My current personal dreams78
![Page 147: Wojciech Seliga wojciech.seliga@spartez.com, T: @wseligas3-eu-west-1.amazonaws.com/presentations2012/36_presentation.pdf · 5/16/12 How to survive automated tests hell Wojciech Seliga](https://reader031.vdocument.in/reader031/viewer/2022013008/5e3af5ab1060d64d7023d879/html5/thumbnails/147.jpg)
Realistic Goals (for us*)
Time Type
2 min for unit tests for 95% of the commits
5 min for base smoke functional tests for 95% of the commits
15 min for ALL tests for 95% of the commits on selected platform
30 min for ALL tests for ALL commits
*My current personal dreams78
![Page 148: Wojciech Seliga wojciech.seliga@spartez.com, T: @wseligas3-eu-west-1.amazonaws.com/presentations2012/36_presentation.pdf · 5/16/12 How to survive automated tests hell Wojciech Seliga](https://reader031.vdocument.in/reader031/viewer/2022013008/5e3af5ab1060d64d7023d879/html5/thumbnails/148.jpg)
Realistic Goals (for us*)
Time Type
2 min for unit tests for 95% of the commits
5 min for base smoke functional tests for 95% of the commits
15 min for ALL tests for 95% of the commits on selected platform
30 min for ALL tests for ALL commits
*My current personal dreams78
![Page 149: Wojciech Seliga wojciech.seliga@spartez.com, T: @wseligas3-eu-west-1.amazonaws.com/presentations2012/36_presentation.pdf · 5/16/12 How to survive automated tests hell Wojciech Seliga](https://reader031.vdocument.in/reader031/viewer/2022013008/5e3af5ab1060d64d7023d879/html5/thumbnails/149.jpg)
Realistic Goals (for us*)
Time Type
2 min for unit tests for 95% of the commits
5 min for base smoke functional tests for 95% of the commits
15 min for ALL tests for 95% of the commits on selected platform
30 min for ALL tests for ALL commits
*My current personal dreams78
![Page 150: Wojciech Seliga wojciech.seliga@spartez.com, T: @wseligas3-eu-west-1.amazonaws.com/presentations2012/36_presentation.pdf · 5/16/12 How to survive automated tests hell Wojciech Seliga](https://reader031.vdocument.in/reader031/viewer/2022013008/5e3af5ab1060d64d7023d879/html5/thumbnails/150.jpg)
Realistic Goals (for us*)
Time Type
2 min for unit tests for 95% of the commits
5 min for base smoke functional tests for 95% of the commits
15 min for ALL tests for 95% of the commits on selected platform
30 min for ALL tests for ALL commits
*My current personal dreams78
![Page 151: Wojciech Seliga wojciech.seliga@spartez.com, T: @wseligas3-eu-west-1.amazonaws.com/presentations2012/36_presentation.pdf · 5/16/12 How to survive automated tests hell Wojciech Seliga](https://reader031.vdocument.in/reader031/viewer/2022013008/5e3af5ab1060d64d7023d879/html5/thumbnails/151.jpg)
Realistic Goals (for us*) p.2
79
![Page 152: Wojciech Seliga wojciech.seliga@spartez.com, T: @wseligas3-eu-west-1.amazonaws.com/presentations2012/36_presentation.pdf · 5/16/12 How to survive automated tests hell Wojciech Seliga](https://reader031.vdocument.in/reader031/viewer/2022013008/5e3af5ab1060d64d7023d879/html5/thumbnails/152.jpg)
Realistic Goals (for us*) p.2
*My current personal dreams79
![Page 153: Wojciech Seliga wojciech.seliga@spartez.com, T: @wseligas3-eu-west-1.amazonaws.com/presentations2012/36_presentation.pdf · 5/16/12 How to survive automated tests hell Wojciech Seliga](https://reader031.vdocument.in/reader031/viewer/2022013008/5e3af5ab1060d64d7023d879/html5/thumbnails/153.jpg)
Realistic Goals (for us*) p.2
Metric Type
>98% green unit tests
>95% green functional tests
<20min average time to fix unit test
<2h average time to fix functional test
*My current personal dreams79
![Page 154: Wojciech Seliga wojciech.seliga@spartez.com, T: @wseligas3-eu-west-1.amazonaws.com/presentations2012/36_presentation.pdf · 5/16/12 How to survive automated tests hell Wojciech Seliga](https://reader031.vdocument.in/reader031/viewer/2022013008/5e3af5ab1060d64d7023d879/html5/thumbnails/154.jpg)
Realistic Goals (for us*) p.2
Metric Type
>98% green unit tests
>95% green functional tests
<20min average time to fix unit test
<2h average time to fix functional test
*My current personal dreams79
![Page 155: Wojciech Seliga wojciech.seliga@spartez.com, T: @wseligas3-eu-west-1.amazonaws.com/presentations2012/36_presentation.pdf · 5/16/12 How to survive automated tests hell Wojciech Seliga](https://reader031.vdocument.in/reader031/viewer/2022013008/5e3af5ab1060d64d7023d879/html5/thumbnails/155.jpg)
Realistic Goals (for us*) p.2
Metric Type
>98% green unit tests
>95% green functional tests
<20min average time to fix unit test
<2h average time to fix functional test
*My current personal dreams79
![Page 156: Wojciech Seliga wojciech.seliga@spartez.com, T: @wseligas3-eu-west-1.amazonaws.com/presentations2012/36_presentation.pdf · 5/16/12 How to survive automated tests hell Wojciech Seliga](https://reader031.vdocument.in/reader031/viewer/2022013008/5e3af5ab1060d64d7023d879/html5/thumbnails/156.jpg)
Realistic Goals (for us*) p.2
Metric Type
>98% green unit tests
>95% green functional tests
<20min average time to fix unit test
<2h average time to fix functional test
*My current personal dreams79
![Page 157: Wojciech Seliga wojciech.seliga@spartez.com, T: @wseligas3-eu-west-1.amazonaws.com/presentations2012/36_presentation.pdf · 5/16/12 How to survive automated tests hell Wojciech Seliga](https://reader031.vdocument.in/reader031/viewer/2022013008/5e3af5ab1060d64d7023d879/html5/thumbnails/157.jpg)
Realistic Goals (for us*) p.2
Metric Type
>98% green unit tests
>95% green functional tests
<20min average time to fix unit test
<2h average time to fix functional test
*My current personal dreams79
![Page 158: Wojciech Seliga wojciech.seliga@spartez.com, T: @wseligas3-eu-west-1.amazonaws.com/presentations2012/36_presentation.pdf · 5/16/12 How to survive automated tests hell Wojciech Seliga](https://reader031.vdocument.in/reader031/viewer/2022013008/5e3af5ab1060d64d7023d879/html5/thumbnails/158.jpg)
Our Possible Future
• Further splitting the code-base, incubation and maturity
• Finer-grained team responsibilities
• Merciless quarantine and purging of flakey tests
• More page objects, less old-school Selenium
• Refactoring/removal of slow tests
• More REST-driven test fixtures and assertions
• More developer's awareness, training, coaching
80
![Page 159: Wojciech Seliga wojciech.seliga@spartez.com, T: @wseligas3-eu-west-1.amazonaws.com/presentations2012/36_presentation.pdf · 5/16/12 How to survive automated tests hell Wojciech Seliga](https://reader031.vdocument.in/reader031/viewer/2022013008/5e3af5ab1060d64d7023d879/html5/thumbnails/159.jpg)
Take-aways
81
![Page 160: Wojciech Seliga wojciech.seliga@spartez.com, T: @wseligas3-eu-west-1.amazonaws.com/presentations2012/36_presentation.pdf · 5/16/12 How to survive automated tests hell Wojciech Seliga](https://reader031.vdocument.in/reader031/viewer/2022013008/5e3af5ab1060d64d7023d879/html5/thumbnails/160.jpg)
Automated testing has cumulative benefits
82
![Page 161: Wojciech Seliga wojciech.seliga@spartez.com, T: @wseligas3-eu-west-1.amazonaws.com/presentations2012/36_presentation.pdf · 5/16/12 How to survive automated tests hell Wojciech Seliga](https://reader031.vdocument.in/reader031/viewer/2022013008/5e3af5ab1060d64d7023d879/html5/thumbnails/161.jpg)
Automated testing has cumulative benefits
...and cumulative cost
82
![Page 162: Wojciech Seliga wojciech.seliga@spartez.com, T: @wseligas3-eu-west-1.amazonaws.com/presentations2012/36_presentation.pdf · 5/16/12 How to survive automated tests hell Wojciech Seliga](https://reader031.vdocument.in/reader031/viewer/2022013008/5e3af5ab1060d64d7023d879/html5/thumbnails/162.jpg)
Splitting codebase and favouring unit tests
are key aspects of short test feedback loop
83
![Page 163: Wojciech Seliga wojciech.seliga@spartez.com, T: @wseligas3-eu-west-1.amazonaws.com/presentations2012/36_presentation.pdf · 5/16/12 How to survive automated tests hell Wojciech Seliga](https://reader031.vdocument.in/reader031/viewer/2022013008/5e3af5ab1060d64d7023d879/html5/thumbnails/163.jpg)
Test Code is Not Trash
84
![Page 164: Wojciech Seliga wojciech.seliga@spartez.com, T: @wseligas3-eu-west-1.amazonaws.com/presentations2012/36_presentation.pdf · 5/16/12 How to survive automated tests hell Wojciech Seliga](https://reader031.vdocument.in/reader031/viewer/2022013008/5e3af5ab1060d64d7023d879/html5/thumbnails/164.jpg)
Test Code is Not Trash
Respect
84
![Page 165: Wojciech Seliga wojciech.seliga@spartez.com, T: @wseligas3-eu-west-1.amazonaws.com/presentations2012/36_presentation.pdf · 5/16/12 How to survive automated tests hell Wojciech Seliga](https://reader031.vdocument.in/reader031/viewer/2022013008/5e3af5ab1060d64d7023d879/html5/thumbnails/165.jpg)
Test Code is Not Trash
DesignRespect
84
![Page 166: Wojciech Seliga wojciech.seliga@spartez.com, T: @wseligas3-eu-west-1.amazonaws.com/presentations2012/36_presentation.pdf · 5/16/12 How to survive automated tests hell Wojciech Seliga](https://reader031.vdocument.in/reader031/viewer/2022013008/5e3af5ab1060d64d7023d879/html5/thumbnails/166.jpg)
Test Code is Not Trash
Design
Maintain
Respect
84
![Page 167: Wojciech Seliga wojciech.seliga@spartez.com, T: @wseligas3-eu-west-1.amazonaws.com/presentations2012/36_presentation.pdf · 5/16/12 How to survive automated tests hell Wojciech Seliga](https://reader031.vdocument.in/reader031/viewer/2022013008/5e3af5ab1060d64d7023d879/html5/thumbnails/167.jpg)
Test Code is Not Trash
Design
MaintainReview
Respect
84
![Page 168: Wojciech Seliga wojciech.seliga@spartez.com, T: @wseligas3-eu-west-1.amazonaws.com/presentations2012/36_presentation.pdf · 5/16/12 How to survive automated tests hell Wojciech Seliga](https://reader031.vdocument.in/reader031/viewer/2022013008/5e3af5ab1060d64d7023d879/html5/thumbnails/168.jpg)
Test Code is Not Trash
Design
MaintainRefactor
Review
Respect
84
![Page 169: Wojciech Seliga wojciech.seliga@spartez.com, T: @wseligas3-eu-west-1.amazonaws.com/presentations2012/36_presentation.pdf · 5/16/12 How to survive automated tests hell Wojciech Seliga](https://reader031.vdocument.in/reader031/viewer/2022013008/5e3af5ab1060d64d7023d879/html5/thumbnails/169.jpg)
Test Code is Not Trash
Design
MaintainRefactor
Review
Respect Restructure
84
![Page 170: Wojciech Seliga wojciech.seliga@spartez.com, T: @wseligas3-eu-west-1.amazonaws.com/presentations2012/36_presentation.pdf · 5/16/12 How to survive automated tests hell Wojciech Seliga](https://reader031.vdocument.in/reader031/viewer/2022013008/5e3af5ab1060d64d7023d879/html5/thumbnails/170.jpg)
Test Code is Not Trash
Design
MaintainRefactor
Share
Review
Respect Restructure
84
![Page 171: Wojciech Seliga wojciech.seliga@spartez.com, T: @wseligas3-eu-west-1.amazonaws.com/presentations2012/36_presentation.pdf · 5/16/12 How to survive automated tests hell Wojciech Seliga](https://reader031.vdocument.in/reader031/viewer/2022013008/5e3af5ab1060d64d7023d879/html5/thumbnails/171.jpg)
Test Code is Not Trash
Design
MaintainRefactor
Share
Review
Respect
Discuss
Restructure
84
![Page 172: Wojciech Seliga wojciech.seliga@spartez.com, T: @wseligas3-eu-west-1.amazonaws.com/presentations2012/36_presentation.pdf · 5/16/12 How to survive automated tests hell Wojciech Seliga](https://reader031.vdocument.in/reader031/viewer/2022013008/5e3af5ab1060d64d7023d879/html5/thumbnails/172.jpg)
Test Code is Not Trash
Design
MaintainRefactor
Share
Review
Prune
Respect
Discuss
Restructure
84
![Page 173: Wojciech Seliga wojciech.seliga@spartez.com, T: @wseligas3-eu-west-1.amazonaws.com/presentations2012/36_presentation.pdf · 5/16/12 How to survive automated tests hell Wojciech Seliga](https://reader031.vdocument.in/reader031/viewer/2022013008/5e3af5ab1060d64d7023d879/html5/thumbnails/173.jpg)
Optimum Balance
85
![Page 174: Wojciech Seliga wojciech.seliga@spartez.com, T: @wseligas3-eu-west-1.amazonaws.com/presentations2012/36_presentation.pdf · 5/16/12 How to survive automated tests hell Wojciech Seliga](https://reader031.vdocument.in/reader031/viewer/2022013008/5e3af5ab1060d64d7023d879/html5/thumbnails/174.jpg)
Optimum Balance
Isolation85
![Page 175: Wojciech Seliga wojciech.seliga@spartez.com, T: @wseligas3-eu-west-1.amazonaws.com/presentations2012/36_presentation.pdf · 5/16/12 How to survive automated tests hell Wojciech Seliga](https://reader031.vdocument.in/reader031/viewer/2022013008/5e3af5ab1060d64d7023d879/html5/thumbnails/175.jpg)
Optimum Balance
Isolation Speed85
![Page 176: Wojciech Seliga wojciech.seliga@spartez.com, T: @wseligas3-eu-west-1.amazonaws.com/presentations2012/36_presentation.pdf · 5/16/12 How to survive automated tests hell Wojciech Seliga](https://reader031.vdocument.in/reader031/viewer/2022013008/5e3af5ab1060d64d7023d879/html5/thumbnails/176.jpg)
Optimum Balance
Isolation Speed Coverage85
![Page 177: Wojciech Seliga wojciech.seliga@spartez.com, T: @wseligas3-eu-west-1.amazonaws.com/presentations2012/36_presentation.pdf · 5/16/12 How to survive automated tests hell Wojciech Seliga](https://reader031.vdocument.in/reader031/viewer/2022013008/5e3af5ab1060d64d7023d879/html5/thumbnails/177.jpg)
Optimum Balance
Isolation Speed Coverage Level85
![Page 178: Wojciech Seliga wojciech.seliga@spartez.com, T: @wseligas3-eu-west-1.amazonaws.com/presentations2012/36_presentation.pdf · 5/16/12 How to survive automated tests hell Wojciech Seliga](https://reader031.vdocument.in/reader031/viewer/2022013008/5e3af5ab1060d64d7023d879/html5/thumbnails/178.jpg)
Optimum Balance
Isolation Speed Coverage Level Structure85
![Page 179: Wojciech Seliga wojciech.seliga@spartez.com, T: @wseligas3-eu-west-1.amazonaws.com/presentations2012/36_presentation.pdf · 5/16/12 How to survive automated tests hell Wojciech Seliga](https://reader031.vdocument.in/reader031/viewer/2022013008/5e3af5ab1060d64d7023d879/html5/thumbnails/179.jpg)
Optimum Balance
Isolation Speed Coverage Level Structure Effort85
![Page 180: Wojciech Seliga wojciech.seliga@spartez.com, T: @wseligas3-eu-west-1.amazonaws.com/presentations2012/36_presentation.pdf · 5/16/12 How to survive automated tests hell Wojciech Seliga](https://reader031.vdocument.in/reader031/viewer/2022013008/5e3af5ab1060d64d7023d879/html5/thumbnails/180.jpg)
Dangerous to temper with
86
![Page 181: Wojciech Seliga wojciech.seliga@spartez.com, T: @wseligas3-eu-west-1.amazonaws.com/presentations2012/36_presentation.pdf · 5/16/12 How to survive automated tests hell Wojciech Seliga](https://reader031.vdocument.in/reader031/viewer/2022013008/5e3af5ab1060d64d7023d879/html5/thumbnails/181.jpg)
Dangerous to temper with
Quality / Determinism86
![Page 182: Wojciech Seliga wojciech.seliga@spartez.com, T: @wseligas3-eu-west-1.amazonaws.com/presentations2012/36_presentation.pdf · 5/16/12 How to survive automated tests hell Wojciech Seliga](https://reader031.vdocument.in/reader031/viewer/2022013008/5e3af5ab1060d64d7023d879/html5/thumbnails/182.jpg)
Dangerous to temper with
MaintainabilityQuality / Determinism86
![Page 183: Wojciech Seliga wojciech.seliga@spartez.com, T: @wseligas3-eu-west-1.amazonaws.com/presentations2012/36_presentation.pdf · 5/16/12 How to survive automated tests hell Wojciech Seliga](https://reader031.vdocument.in/reader031/viewer/2022013008/5e3af5ab1060d64d7023d879/html5/thumbnails/183.jpg)
87
![Page 184: Wojciech Seliga wojciech.seliga@spartez.com, T: @wseligas3-eu-west-1.amazonaws.com/presentations2012/36_presentation.pdf · 5/16/12 How to survive automated tests hell Wojciech Seliga](https://reader031.vdocument.in/reader031/viewer/2022013008/5e3af5ab1060d64d7023d879/html5/thumbnails/184.jpg)
There are no universal rules - silver bullets
87
![Page 185: Wojciech Seliga wojciech.seliga@spartez.com, T: @wseligas3-eu-west-1.amazonaws.com/presentations2012/36_presentation.pdf · 5/16/12 How to survive automated tests hell Wojciech Seliga](https://reader031.vdocument.in/reader031/viewer/2022013008/5e3af5ab1060d64d7023d879/html5/thumbnails/185.jpg)
There are no universal rules - silver bullets
We are expected to find optimum balance for our
specific case
87
![Page 186: Wojciech Seliga wojciech.seliga@spartez.com, T: @wseligas3-eu-west-1.amazonaws.com/presentations2012/36_presentation.pdf · 5/16/12 How to survive automated tests hell Wojciech Seliga](https://reader031.vdocument.in/reader031/viewer/2022013008/5e3af5ab1060d64d7023d879/html5/thumbnails/186.jpg)
There are no universal rules - silver bullets
We are expected to find optimum balance for our
specific case
Definition of “optimum” constantly changes
87
![Page 187: Wojciech Seliga wojciech.seliga@spartez.com, T: @wseligas3-eu-west-1.amazonaws.com/presentations2012/36_presentation.pdf · 5/16/12 How to survive automated tests hell Wojciech Seliga](https://reader031.vdocument.in/reader031/viewer/2022013008/5e3af5ab1060d64d7023d879/html5/thumbnails/187.jpg)
Otherwise
88
![Page 188: Wojciech Seliga wojciech.seliga@spartez.com, T: @wseligas3-eu-west-1.amazonaws.com/presentations2012/36_presentation.pdf · 5/16/12 How to survive automated tests hell Wojciech Seliga](https://reader031.vdocument.in/reader031/viewer/2022013008/5e3af5ab1060d64d7023d879/html5/thumbnails/188.jpg)
Otherwise
89
![Page 189: Wojciech Seliga wojciech.seliga@spartez.com, T: @wseligas3-eu-west-1.amazonaws.com/presentations2012/36_presentation.pdf · 5/16/12 How to survive automated tests hell Wojciech Seliga](https://reader031.vdocument.in/reader031/viewer/2022013008/5e3af5ab1060d64d7023d879/html5/thumbnails/189.jpg)
Did I mention that Page Objects pattern?
90
![Page 190: Wojciech Seliga wojciech.seliga@spartez.com, T: @wseligas3-eu-west-1.amazonaws.com/presentations2012/36_presentation.pdf · 5/16/12 How to survive automated tests hell Wojciech Seliga](https://reader031.vdocument.in/reader031/viewer/2022013008/5e3af5ab1060d64d7023d879/html5/thumbnails/190.jpg)
We are hiring in Gdańsk!
Java devs Javascript devs UI/UX designers
Team leader
Do you want to help us?
91
![Page 191: Wojciech Seliga wojciech.seliga@spartez.com, T: @wseligas3-eu-west-1.amazonaws.com/presentations2012/36_presentation.pdf · 5/16/12 How to survive automated tests hell Wojciech Seliga](https://reader031.vdocument.in/reader031/viewer/2022013008/5e3af5ab1060d64d7023d879/html5/thumbnails/191.jpg)
Credits
• Photos:• http://www.flickr.com/photos/toofarnorth/ - Dragon
• http://www.flickr.com/photos/striatic - Frustration
• http://www.flickr.com/photos/leeadlaf/ - Broken window
• http://www.flickr.com/photos/johnloo/ - Lightbulb
• Dariusz Kordoński - for Test Improvements Leadership in JIRA
92
![Page 192: Wojciech Seliga wojciech.seliga@spartez.com, T: @wseligas3-eu-west-1.amazonaws.com/presentations2012/36_presentation.pdf · 5/16/12 How to survive automated tests hell Wojciech Seliga](https://reader031.vdocument.in/reader031/viewer/2022013008/5e3af5ab1060d64d7023d879/html5/thumbnails/192.jpg)
Thank You
93