zero to tested
TRANSCRIPT
Zero to Tested “Bullet proof test automation in hostile environments.”
The test automation promised land
Small team of full stack developers (with specialists in specific areas)
BDD/TDD
Strong adherence to the Test Pyramid
Isolation of different layers (good use of stubs, mocks, fakes)
Thin layer of e2e tests
CONTINUOUS DELIVERY!
For most projects
Fairly unrealistic, at least not anytime soon...
But this means...
“Often reliant on end to end automated tests which have a terrible reputation!”
Why?
People acknowledge e2e tests are essential
Tests are flaky
CI infrastructure is poorly managed (configuration drift)
Development of tests lags behind the development cycle (“test
sprints”)
Often ignored by most of the team
Wallboard is more often RED than GREEN
Worth reading...
http://googletesting.blogspot.co.uk/2015/04/just-say-no-to-more-end-to-end-tests.html
http://www.alwaysagileconsulting.com/articles/end-to-end-testing-considered-harmful/
http://devblog.songkick.com/2012/07/16/from-15-hours-to-15-seconds-reducing-a-crushing-build
-time/
But I prefer
https://www.symphonious.net/2015/04/30/making-end-to-end-tests-work/
10,000 end to end acceptance tests, primary form of quality control
complete in 50 minutes, including deploying/starting and tearing down the
servers
all required to pass before we consider a version releasable
included in our information radiators to ensure the team has constant visibility
are owned by the whole team
My theory.
Us “test automation” folk have let ourselves down
Even if we are not in an agile utopia…
Have we exhausted every avenue to make tests reliable?
Need to prove we are doing everything we can to write bulletproof
tests
Now for the practical stuff!Even with minimal influence on budget/project direction we can usually hugely improve on:
CI Infrastructure automation
Automation code and architecture
Let’s take a look at a thin vertical slice through a solid automation
setup...
CI Infrastructure automation
Infrastructure automationStart from an empty EC2 account and finish with:
Fully functioning Jenkins instance
Automated tests running successfully against a “flaky” test
application
Deployed in < 5 mins, very little code, fully automated and
“Ephemeral”
DevOps Toys - Chef, Packer, Terraform, AWS, Docker
BAKE SPIN UP RUN TESTS
Chef
Packer
DevOps Toys - Chef, Packer, Terraform, AWS, Docker
BAKE SPIN UP RUN TESTS
Terraform
DevOps Toys - Chef, Packer, Terraform, AWS, Docker
BAKE SPIN UP RUN TESTS
Jenkins Job DSL Plugin
Docker
Docker
Has it finished?
Recap...
Automated provisioning of AWS infrastructure
Automated build of ‘pre-baked’ AMI
Automatically provisioned Jenkins with Docker
Docker allows clients to have complete control over dependencies
Job DSL for building jobs
Take awayEverything is in source control, no more snowflake CI boxes, no more
handcrafted Jenkins jobs.
“But it’s not scalable!” - switch to agent model with auto scaled
agents
“But it’s not secure!”- not at the moment, but easy to add e.g. github
auth
More importantly It’s ephemeral - you can destroy if compromised
Automation Code & Architecture
Tooling choiceIn Ruby Land the “go-to” libraries are:
Capybara
Selenium Webdriver
Poltergeist/PhantomJS
SitePrism
RSpec
Automation Code
What are the biggest failings of most end to end regression test suites?
My thoughts
Inappropriate synchronisation strategies (page load, async JS)
Inability to deal with non-deterministic applications and environments
Inability to deal with “eventual consistency”
Poor reporting of failure when it occurs
RETRY ALL THE THINGS!
A flaky app...
Page Load
Handled by Selenium Webdriver!
Async JSAvoid this...
Async JSGood libraries have first class support for this:
VisibilityElements exist in DOM but CSS means they are not visible.
Good libraries have first class support for this:
VisibilityWorth remembering…
● display:none
● visibility:hidden
● Opacity:0
● Dimensions
And remember there is a W3C spec for Webdriver:
https://w3c.github.io/webdriver/webdriver-spec.html#dfn-element-displayed
Non determinism
Should we be expected to deal with this?
Maybe, maybe not
Either way don’t just give up, at least detect, log and report on it!
Non determinismDon’t just rely on re-run - be proactive and report on failure
Libraries probably won’t help here!
Eventual Consistency
Caching, load balancing, asynchronous architectures etc.
TL;DR - You might need to wait a bit...
Eventual ConsistencyYou might have to ‘roll your own’ wait helpers from time to time
Logging
Configurable levels: debug, info, warn, error etc
Pretty colours - makes debugging broken builds easier
Have log aggregation for your application? (RSyslog, Logstash,
Kibana)
Now you can combine test and application logs for super debugging
power!
Test the tests
We should unit test any “library code” that has significant complexity.
Static Analysis
Helps maintain consistency, best practice and readability of code.
Reporting
Reporting
Wrapping up
We need to up our game when writing end to end tests
Make the provision of CI infrastructure automated and ephemeral
Write automation code that can handle hostile conditions
Report on your findings to effect change
Questions?