![Page 2: Escaping Test Hell - Our Journey - XPDays Ukraine 2013](https://reader034.vdocument.in/reader034/viewer/2022042714/554a13e5b4c905825d8b4ce2/html5/thumbnails/2.jpg)
About me
• Coding for 30 years, now only in "free time"
• Agile Practices (inc. TDD) since 2003
• Dev Nerd, Tech Leader, Agile Coach, Speaker, PHB
• 6 years with Atlassian (JIRA Dev Manager)
• Spartez Co-founder & CEO
![Page 3: Escaping Test Hell - Our Journey - XPDays Ukraine 2013](https://reader034.vdocument.in/reader034/viewer/2022042714/554a13e5b4c905825d8b4ce2/html5/thumbnails/3.jpg)
XP PromiseC
ost
of C
hang
e
Time
WaterfallXP
![Page 4: Escaping Test Hell - Our Journey - XPDays Ukraine 2013](https://reader034.vdocument.in/reader034/viewer/2022042714/554a13e5b4c905825d8b4ce2/html5/thumbnails/4.jpg)
1.5 year ago
![Page 5: Escaping Test Hell - Our Journey - XPDays Ukraine 2013](https://reader034.vdocument.in/reader034/viewer/2022042714/554a13e5b4c905825d8b4ce2/html5/thumbnails/5.jpg)
Almost 10 years of accumulating
garbage automatic tests
![Page 6: Escaping Test Hell - Our Journey - XPDays Ukraine 2013](https://reader034.vdocument.in/reader034/viewer/2022042714/554a13e5b4c905825d8b4ce2/html5/thumbnails/6.jpg)
About 20 000 tests on all levels of abstraction
![Page 7: Escaping Test Hell - Our Journey - XPDays Ukraine 2013](https://reader034.vdocument.in/reader034/viewer/2022042714/554a13e5b4c905825d8b4ce2/html5/thumbnails/7.jpg)
Very slow (even hours)and fragile feedback loop
![Page 8: Escaping Test Hell - Our Journey - XPDays Ukraine 2013](https://reader034.vdocument.in/reader034/viewer/2022042714/554a13e5b4c905825d8b4ce2/html5/thumbnails/8.jpg)
Serious performance and reliability issues
![Page 9: Escaping Test Hell - Our Journey - XPDays Ukraine 2013](https://reader034.vdocument.in/reader034/viewer/2022042714/554a13e5b4c905825d8b4ce2/html5/thumbnails/9.jpg)
Dispirited devs accepting RED as a norm
![Page 10: Escaping Test Hell - Our Journey - XPDays Ukraine 2013](https://reader034.vdocument.in/reader034/viewer/2022042714/554a13e5b4c905825d8b4ce2/html5/thumbnails/10.jpg)
FeedbackSpeed
`Test
Quality
![Page 11: Escaping Test Hell - Our Journey - XPDays Ukraine 2013](https://reader034.vdocument.in/reader034/viewer/2022042714/554a13e5b4c905825d8b4ce2/html5/thumbnails/11.jpg)
Test Code is Not Trash
Design
MaintainRefactor
Share
Review
Prune
Respect
Discuss
Restructure
![Page 12: Escaping Test Hell - Our Journey - XPDays Ukraine 2013](https://reader034.vdocument.in/reader034/viewer/2022042714/554a13e5b4c905825d8b4ce2/html5/thumbnails/12.jpg)
Test Pyramid
Unit Tests (including QUnit)
REST / HTML Tests
Selenium
![Page 13: Escaping Test Hell - Our Journey - XPDays Ukraine 2013](https://reader034.vdocument.in/reader034/viewer/2022042714/554a13e5b4c905825d8b4ce2/html5/thumbnails/13.jpg)
Optimum Balance
Isolation Speed Coverage Level Access Effort
![Page 14: Escaping Test Hell - Our Journey - XPDays Ukraine 2013](https://reader034.vdocument.in/reader034/viewer/2022042714/554a13e5b4c905825d8b4ce2/html5/thumbnails/14.jpg)
Dangerous to temper with
MaintainabilityQuality / Determinism
![Page 15: Escaping Test Hell - Our Journey - XPDays Ukraine 2013](https://reader034.vdocument.in/reader034/viewer/2022042714/554a13e5b4c905825d8b4ce2/html5/thumbnails/15.jpg)
Now
![Page 16: Escaping Test Hell - Our Journey - XPDays Ukraine 2013](https://reader034.vdocument.in/reader034/viewer/2022042714/554a13e5b4c905825d8b4ce2/html5/thumbnails/16.jpg)
People - MotivationMaking GREEN the norm
![Page 17: Escaping Test Hell - Our Journey - XPDays Ukraine 2013](https://reader034.vdocument.in/reader034/viewer/2022042714/554a13e5b4c905825d8b4ce2/html5/thumbnails/17.jpg)
Shades of RedShades of Green
![Page 18: Escaping Test Hell - Our Journey - XPDays Ukraine 2013](https://reader034.vdocument.in/reader034/viewer/2022042714/554a13e5b4c905825d8b4ce2/html5/thumbnails/18.jpg)
Pragmatic CI Health
![Page 19: Escaping Test Hell - Our Journey - XPDays Ukraine 2013](https://reader034.vdocument.in/reader034/viewer/2022042714/554a13e5b4c905825d8b4ce2/html5/thumbnails/19.jpg)
Build Tiers and Policy
Tier A1 - green soon after all commits
Tier A2 - green at the end of the day
Tier A3 - green at the end of the iteration
unit tests and functional* tests
WebDriver and bundled plugins tests
supported platforms tests, compatibility tests
![Page 20: Escaping Test Hell - Our Journey - XPDays Ukraine 2013](https://reader034.vdocument.in/reader034/viewer/2022042714/554a13e5b4c905825d8b4ce2/html5/thumbnails/20.jpg)
Wallboards: Constant
Awareness
![Page 21: Escaping Test Hell - Our Journey - XPDays Ukraine 2013](https://reader034.vdocument.in/reader034/viewer/2022042714/554a13e5b4c905825d8b4ce2/html5/thumbnails/21.jpg)
Extensive Training
• assertThat over assertTrue/False and assertEquals
• avoiding races - Atlassian Selenium with its TimedElement
• Favouring unit tests over functional tests
• Promoting Page Objects
• Brownbags, blogs, code reviews
![Page 22: Escaping Test Hell - Our Journey - XPDays Ukraine 2013](https://reader034.vdocument.in/reader034/viewer/2022042714/554a13e5b4c905825d8b4ce2/html5/thumbnails/22.jpg)
Quality
![Page 23: Escaping Test Hell - Our Journey - XPDays Ukraine 2013](https://reader034.vdocument.in/reader034/viewer/2022042714/554a13e5b4c905825d8b4ce2/html5/thumbnails/23.jpg)
Automatic Flakiness Detection Quarantine
Re-run failed tests and see if they pass
![Page 24: Escaping Test Hell - Our Journey - XPDays Ukraine 2013](https://reader034.vdocument.in/reader034/viewer/2022042714/554a13e5b4c905825d8b4ce2/html5/thumbnails/24.jpg)
Quarantine - Healing
![Page 25: Escaping Test Hell - Our Journey - XPDays Ukraine 2013](https://reader034.vdocument.in/reader034/viewer/2022042714/554a13e5b4c905825d8b4ce2/html5/thumbnails/25.jpg)
SlowMo - expose races
![Page 26: Escaping Test Hell - Our Journey - XPDays Ukraine 2013](https://reader034.vdocument.in/reader034/viewer/2022042714/554a13e5b4c905825d8b4ce2/html5/thumbnails/26.jpg)
Selenium 1
![Page 27: Escaping Test Hell - Our Journey - XPDays Ukraine 2013](https://reader034.vdocument.in/reader034/viewer/2022042714/554a13e5b4c905825d8b4ce2/html5/thumbnails/27.jpg)
Selenium ditching Sky did not fall in
![Page 28: Escaping Test Hell - Our Journey - XPDays Ukraine 2013](https://reader034.vdocument.in/reader034/viewer/2022042714/554a13e5b4c905825d8b4ce2/html5/thumbnails/28.jpg)
Ditching - benefits
• Freed build agents - better system throughput
• Boosted morale
• Gazillion of developer hours saved
• Money saved on infrastructure
![Page 29: Escaping Test Hell - Our Journey - XPDays Ukraine 2013](https://reader034.vdocument.in/reader034/viewer/2022042714/554a13e5b4c905825d8b4ce2/html5/thumbnails/29.jpg)
Ditching - due diligence
• conducting the audit - analysis of the coverage we lost
• determining which tests needs to rewritten (e.g. security related)
• rewriting some of the tests (good job for new hires + a senior mentor)
![Page 30: Escaping Test Hell - Our Journey - XPDays Ukraine 2013](https://reader034.vdocument.in/reader034/viewer/2022042714/554a13e5b4c905825d8b4ce2/html5/thumbnails/30.jpg)
Flaky Browser-based TestsRaces between test code and asynchronous page logic
Playing with "loading" CSS class does not really help
![Page 31: Escaping Test Hell - Our Journey - XPDays Ukraine 2013](https://reader034.vdocument.in/reader034/viewer/2022042714/554a13e5b4c905825d8b4ce2/html5/thumbnails/31.jpg)
Races Removal with Tracing// in the browser:function mySearchClickHandler() { doSomeXhr().always(function() { // This executes when the XHR has completed (either success or failure) JIRA.trace("search.completed"); });}// In production code JIRA.trace is a no-op
// in my page object:@InjectTraceContext traceContext; public SearchResults doASearch() { Tracer snapshot = traceContext.checkpoint(); getSearchButton().click(); // causes mySearchClickHandler to be invoked // This waits until the "search.completed" // event has been emitted, *after* previous snapshot traceContext.waitFor(snapshot, "search.completed"); return pageBinder.bind(SearchResults.class);}
![Page 32: Escaping Test Hell - Our Journey - XPDays Ukraine 2013](https://reader034.vdocument.in/reader034/viewer/2022042714/554a13e5b4c905825d8b4ce2/html5/thumbnails/32.jpg)
Can we halve our build times?
Speed
![Page 33: Escaping Test Hell - Our Journey - XPDays Ukraine 2013](https://reader034.vdocument.in/reader034/viewer/2022042714/554a13e5b4c905825d8b4ce2/html5/thumbnails/33.jpg)
Parallel Execution - Theory
End of Build
Batches
Start of Build Time
![Page 34: Escaping Test Hell - Our Journey - XPDays Ukraine 2013](https://reader034.vdocument.in/reader034/viewer/2022042714/554a13e5b4c905825d8b4ce2/html5/thumbnails/34.jpg)
Parallel Execution
End of Build
Batches
Start of Build Time
![Page 35: Escaping Test Hell - Our Journey - XPDays Ukraine 2013](https://reader034.vdocument.in/reader034/viewer/2022042714/554a13e5b4c905825d8b4ce2/html5/thumbnails/35.jpg)
Parallel Execution - Reality Bites
End of Build
A1
Batches
Start of Build
Agent availability
Time
![Page 36: Escaping Test Hell - Our Journey - XPDays Ukraine 2013](https://reader034.vdocument.in/reader034/viewer/2022042714/554a13e5b4c905825d8b4ce2/html5/thumbnails/36.jpg)
Dynamic Test Execution Dispatch - Hallelujah
![Page 37: Escaping Test Hell - Our Journey - XPDays Ukraine 2013](https://reader034.vdocument.in/reader034/viewer/2022042714/554a13e5b4c905825d8b4ce2/html5/thumbnails/37.jpg)
"You can't manage what you can't measure."
not by W. Edwards Deming
If you believe just in it
you are doomed.
![Page 38: Escaping Test Hell - Our Journey - XPDays Ukraine 2013](https://reader034.vdocument.in/reader034/viewer/2022042714/554a13e5b4c905825d8b4ce2/html5/thumbnails/38.jpg)
You can't improve something if you can't measure it
Profiler, Build statistics, Logs, statsd → Graphite
![Page 39: Escaping Test Hell - Our Journey - XPDays Ukraine 2013](https://reader034.vdocument.in/reader034/viewer/2022042714/554a13e5b4c905825d8b4ce2/html5/thumbnails/39.jpg)
Anatomy of Build*
CompilationPackaging
Executing Tests
Fetching Dependencies
*Any resemblance to maven build is entirely accidental
SCM Update
Agent Availability/Setup
Publishing Results
![Page 40: Escaping Test Hell - Our Journey - XPDays Ukraine 2013](https://reader034.vdocument.in/reader034/viewer/2022042714/554a13e5b4c905825d8b4ce2/html5/thumbnails/40.jpg)
JIRA Unit Tests Build
Compilation (7min)
Packaging (0min)
Executing Tests (7min)Fetching Dependencies (1.5min)
SCM Update (2min)
Agent Availability/Setup (mean 10min)
Publishing Results (1min)
![Page 41: Escaping Test Hell - Our Journey - XPDays Ukraine 2013](https://reader034.vdocument.in/reader034/viewer/2022042714/554a13e5b4c905825d8b4ce2/html5/thumbnails/41.jpg)
Decreasing Test Execution Time
to ZERRO alone would not let us
achieve our goal!
![Page 42: Escaping Test Hell - Our Journey - XPDays Ukraine 2013](https://reader034.vdocument.in/reader034/viewer/2022042714/554a13e5b4c905825d8b4ce2/html5/thumbnails/42.jpg)
Agent Availability/Setup
• starved builds due to busy agents building very long builds
• time synchronization issue - NTPD problem
Fixes applied
![Page 43: Escaping Test Hell - Our Journey - XPDays Ukraine 2013](https://reader034.vdocument.in/reader034/viewer/2022042714/554a13e5b4c905825d8b4ce2/html5/thumbnails/43.jpg)
• Proximity of SCM repo
• shallow git clones are not so fast and lightweight + generating extra git server CPU load
• git clone per agent/plan + git pull + git clone per build (hard links!)
• Atlassian Stash was thankful (queue)
SCM Update - Checkout time
2 min → 5 seconds
Trade disk space for speed
![Page 44: Escaping Test Hell - Our Journey - XPDays Ukraine 2013](https://reader034.vdocument.in/reader034/viewer/2022042714/554a13e5b4c905825d8b4ce2/html5/thumbnails/44.jpg)
![Page 45: Escaping Test Hell - Our Journey - XPDays Ukraine 2013](https://reader034.vdocument.in/reader034/viewer/2022042714/554a13e5b4c905825d8b4ce2/html5/thumbnails/45.jpg)
• Fix Predator
• Sandboxing/isolation agent trade-off:rm -rf $HOME/.m2/repository/com/atlassian/*
intofind $HOME/.m2/repository/com/atlassian/ -name “*SNAPSHOT*” | xargs rm
• Network hardware failure found (dropping packets)
Fetching Dependencies
1.5 min → 10 seconds
![Page 46: Escaping Test Hell - Our Journey - XPDays Ukraine 2013](https://reader034.vdocument.in/reader034/viewer/2022042714/554a13e5b4c905825d8b4ce2/html5/thumbnails/46.jpg)
Compilation
• Restructuring multi-pom maven project and dependencies
• Maven 3 parallel compilation FTW -T 1.5C*optimal factor thanks to scientific trial and error research
7 min → 1 min
![Page 47: Escaping Test Hell - Our Journey - XPDays Ukraine 2013](https://reader034.vdocument.in/reader034/viewer/2022042714/554a13e5b4c905825d8b4ce2/html5/thumbnails/47.jpg)
Unit Test Execution
• Splitting unit tests into 2 buckets: good and legacy (much longer)
• Maven 3 parallel test execution (-T 1.5C)
7 min → 5 min
3000 poor tests(5min)
11000 good tests(1.5min)
![Page 48: Escaping Test Hell - Our Journey - XPDays Ukraine 2013](https://reader034.vdocument.in/reader034/viewer/2022042714/554a13e5b4c905825d8b4ce2/html5/thumbnails/48.jpg)
Functional Tests
• Selenium 1 removal did help
• Faster reset/restore (avoid unnecessary stuff, intercepting SQL operations for debug purposes - building stacktraces is costly)
• Restoring via Backdoor REST API
• Using REST API for common setup/teardown operations
![Page 49: Escaping Test Hell - Our Journey - XPDays Ukraine 2013](https://reader034.vdocument.in/reader034/viewer/2022042714/554a13e5b4c905825d8b4ce2/html5/thumbnails/49.jpg)
Functional Tests
We like this trend
![Page 50: Escaping Test Hell - Our Journey - XPDays Ukraine 2013](https://reader034.vdocument.in/reader034/viewer/2022042714/554a13e5b4c905825d8b4ce2/html5/thumbnails/50.jpg)
Publishing Results
• Server log allocation per test → using now Backdoor REST API (was Selenium)
• Bamboo DB performance degradation for rich build history - to be addressed
1 min → 40 s
![Page 51: Escaping Test Hell - Our Journey - XPDays Ukraine 2013](https://reader034.vdocument.in/reader034/viewer/2022042714/554a13e5b4c905825d8b4ce2/html5/thumbnails/51.jpg)
Unexpected Problem
• Stability issues with our CI server
• The bottleneck changed from I/O to CPU
• Too many agents per physical machine
![Page 52: Escaping Test Hell - Our Journey - XPDays Ukraine 2013](https://reader034.vdocument.in/reader034/viewer/2022042714/554a13e5b4c905825d8b4ce2/html5/thumbnails/52.jpg)
JIRA Unit Tests Build Improved
Compilation (1min)
Packaging (0min)
Executing Tests (5min)
Fetching Dependencies (10sec)
SCM Update (5sec)
Agent Availability/Setup (3min)*
Publishing Results (40sec)
![Page 53: Escaping Test Hell - Our Journey - XPDays Ukraine 2013](https://reader034.vdocument.in/reader034/viewer/2022042714/554a13e5b4c905825d8b4ce2/html5/thumbnails/53.jpg)
Improvements Summary
Tests Before After Improvement %
Unit tests 29 min 17 min 41%
Functional tests 56 min 34 min 39%
WebDriver tests 39 min 21 min 46%
Overall 124 min 72 min 42%
* Additional ca. 5% improvement expected once new git clone strategy is consistently rolled-out everywhere
![Page 54: Escaping Test Hell - Our Journey - XPDays Ukraine 2013](https://reader034.vdocument.in/reader034/viewer/2022042714/554a13e5b4c905825d8b4ce2/html5/thumbnails/54.jpg)
The Quality Follows
![Page 55: Escaping Test Hell - Our Journey - XPDays Ukraine 2013](https://reader034.vdocument.in/reader034/viewer/2022042714/554a13e5b4c905825d8b4ce2/html5/thumbnails/55.jpg)
But that's still bad
We want CI feedback loop in a few minutes maximum
![Page 56: Escaping Test Hell - Our Journey - XPDays Ukraine 2013](https://reader034.vdocument.in/reader034/viewer/2022042714/554a13e5b4c905825d8b4ce2/html5/thumbnails/56.jpg)
Splitting The Codebase
![Page 57: Escaping Test Hell - Our Journey - XPDays Ukraine 2013](https://reader034.vdocument.in/reader034/viewer/2022042714/554a13e5b4c905825d8b4ce2/html5/thumbnails/57.jpg)
Codebase Split - Problems
• Organizational concerns - understanding, managing, integrating, releasing
• Mindset change - if something worked for 10 years why to change it?
• We damned ourselves with big buckets for all tests - where do they belong to?
![Page 58: Escaping Test Hell - Our Journey - XPDays Ukraine 2013](https://reader034.vdocument.in/reader034/viewer/2022042714/554a13e5b4c905825d8b4ce2/html5/thumbnails/58.jpg)
Splitting code base• Step 0 - JIRA Importers Plugin (3.5 years ago)
• Step 1- New Issue View and NavigatorJIRA 6.0
![Page 59: Escaping Test Hell - Our Journey - XPDays Ukraine 2013](https://reader034.vdocument.in/reader034/viewer/2022042714/554a13e5b4c905825d8b4ce2/html5/thumbnails/59.jpg)
We are still escaping hell. Hell sucks in your soul.
![Page 60: Escaping Test Hell - Our Journey - XPDays Ukraine 2013](https://reader034.vdocument.in/reader034/viewer/2022042714/554a13e5b4c905825d8b4ce2/html5/thumbnails/60.jpg)
Conclusions
• Visibility and problem awareness help
• Maintaing huge testbed is difficult and costly
• Measure the problem, measure improvements
• No prejudice - no sacred cows
• Automated tests are not one-off investment, it's a continuous journey
• Performance is a damn important feature
![Page 61: Escaping Test Hell - Our Journey - XPDays Ukraine 2013](https://reader034.vdocument.in/reader034/viewer/2022042714/554a13e5b4c905825d8b4ce2/html5/thumbnails/61.jpg)
Revised XP PromiseC
ost
of C
hang
e
Time
WaterfallXPSad Reality
![Page 62: Escaping Test Hell - Our Journey - XPDays Ukraine 2013](https://reader034.vdocument.in/reader034/viewer/2022042714/554a13e5b4c905825d8b4ce2/html5/thumbnails/62.jpg)
Interested in such stuff?
Talk to me at the conference or visit http://www.spartez.com/careers
We are hiring in Gdańsk
![Page 63: Escaping Test Hell - Our Journey - XPDays Ukraine 2013](https://reader034.vdocument.in/reader034/viewer/2022042714/554a13e5b4c905825d8b4ce2/html5/thumbnails/63.jpg)
• Turtle - by Jonathan Zander, CC-BY-SA-3.0
• Loading - by MatthewJ13, CC-SA-3.0
• Magic Potion - by Koolmann1, CC-BY-SA-2.0
• Merlin Tool - by By L. Mahin, CC-BY-SA-3.0
• Choose Pills - by *rockysprings, CC-BY-SA-3.0
• Flashing Red Light - bt Chris Phan, CC BY 2.0
Images - Credits
![Page 64: Escaping Test Hell - Our Journey - XPDays Ukraine 2013](https://reader034.vdocument.in/reader034/viewer/2022042714/554a13e5b4c905825d8b4ce2/html5/thumbnails/64.jpg)
Thank You!
Tweet your feedback at @wseliga