stop being lazy and test your software

Stop Being Lazy and Test Your Software!

Laura FrankSoftware Engineer, Codeship

0.7.0

ImageLayers

Panamax

Berlin

Agenda

Continuous Delivery

Why Do We Test?

Testing with Docker Compose

Faster Testing with Docker

Why Do We Test?Motivations and frustrations

Testing software has been a practice since the very first machines were used

for computing.

The women who programmed the ENIAC to

calculate trajectories periodically checked the

results with a correct, hand-computed table.

working software in production

testing

code reviews

version control

pair programming

high-availability architecture

MotivatorsUpdate application without breaking things!

Validate functionality of updates

Be able to trust deployment checks (in CI/CD workflow)

Confirm that refactoring didn’t break existing functionality

Why do you skip tests?

45%

15%5%

35% Too slow to run suite 35%

Annoying extra step 15%

Don’t see the point 5%

Slows down deployment 45%

10

FrustratorsIt takes me an hour to write a single test

My new tests just duplicate existing coverages so there’s no point (integration overlapping with unit)

My suite fails all the time for no apparent reason…?

It takes my test suite over 20 minutes to run, so I don’t run it locally

Testing distracts me from my normal development workflow

Setting up a testing environment is a huge pain

It takes too long to deploy when I have a huge test suite

FrustratorsIt takes me an hour to write a single test

My new tests just duplicate existing coverages so there’s no point (integration overlapping with unit)

My suite fails all the time for no apparent reason…?

It takes my test suite over 20 minutes to run, so I don’t run it locally

Testing distracts me from my normal development workflow

Setting up a testing environment is a huge pain

It takes too long to deploy when I have a huge test suite

Using Docker can alleviate some frustrations

associated with testing.

Testing with DockerComposeIt’s easy, I promise

Docker and testing are a great pair.

❌✔

With Docker Compose, it is incredibly easy to create consistent, reproducible environments.

❌✔

Many of us already useDocker Compose to setup dev environments.

…But we stop there.

Use your Docker Composeenvironment for testing

as well.

app: build: . command: rails server -p 3000 -b '0.0.0.0'volumes: - .:/app

ports: - '3000:3000'

links: - postgres

postgres:image: postgres:9.4

A simple Rails app

?!

git clone && docker-compose up

hackity hack

How do I make tests happen?

Development workflow

app: build: . command: rails server -p 3000 -b '0.0.0.0'volumes: - .:/app

ports: - '3000:3000'

links: - postgres

postgres:image: postgres:9.4

A simple Rails app

app: build: . command: rspec specvolumes: - .:/app

ports: - '3000:3000'

links: - postgres

postgres:image: postgres:9.4

A simple Rails app

In most cases, we need to do a bit of setup

before running tests.

You might even need a different Dockerfile.

Docker Compose makes this easy

app: build: . dockerfile: Dockerfile.testcommand: rake testvolumes:

- .:/app

🚨 A word of warning! 🚨

🚨 Race conditions! 🚨

You can also run one-off commands against a service

with Docker Compose.

docker-compose run -e "RAILS_ENV=test" \app rake db:setup

docker-compose run -e "RAILS_ENV=test" \app test-command path/to/spec.rb

docker-compose up #-d if you wish

Usual Docker Compose development environment

Run rake task to set up db

Then run tests against Docker services

Additional Docker Compose Testing Config Options

environment:RACK_ENV: test

Additional Docker Compose Testing Config Options

$ docker-compose up -d $ ./run_tests$ docker-compose stop $ docker-compose rm –f

Docker delivers apredictable, reproducible

testing environment. Period.

Continuous Integration, Testing, and DockerFail fast and not in production!

Continuous integration and testing go hand in hand.

👯

Relying on CI/CD withoutadequate test coverage

is not a great idea.

Would you rather…

✔

💩Find out your code is broken by seeing a failed run of your CI/CD system?

Find out your code is broken by seeing 500s on 500s on 500s in production?

What about running tests like this…

…inside a container?

We run our unit tests in a conatiner during development,

so we should do that during deployment too, right?

And if we’re running our services in containers during development,

they should be running in containers in

production, right?

🚨 A word of warning! 🚨

—Jérôme Petazzoni

“Docker-in-Docker is not 100% made of sparkles, ponies, and unicorns.

46

Docker in Docker

CAUTION!YMMVSee https://jpetazzo.github.io

- Mixing file systems == bad time- Shared build cache == bad time- Lots of other stuff == bad time

Shared daemon – what we really want

We get this by binding the Docker socket.

Parallel Testing with DockerBreak it up.

Why do you skip tests?

45%

15%5%

35% Too slow to run suite 35%

Annoying extra step 15%

Don’t see the point 5%

Slows down deployment 45%

52

When developing, it’s easy to think of a container as

a small VM that runs a specific workload.

✔

But if we change our thinkingto treat containers as just processes, then we can do

some pretty cool stuff.

✔

A syntax similar to Docker Compose forservice definition…

57

web:build:image: my_appdockerfile_path: Dockerfile

links:- redis- postgres

redis:image: redis

postgres:image: postgres

And use it to also define testing steps…

58

- type: parallelsteps:- service: democommand: ruby check.rb

- service: democommand: rake teaspoon suite=jasmine

- service: democommand: rake test

- type: parallelsteps:

- service: democommand: some-test-command

- service: democommand: another-test-command

- service: democommand: yet-another-test-command

Each step is run independently in a container

59

- service: democommand: some-test-command

- service: democommand: another-test-command

- service: democommand: yet-another-test-command

And each container may be located in a range of availability zones

60

Test compose context

Test runner

jettest-1

test-0

test-3

Docker makes this possible byproviding the means to create

a predictable, reproducibletesting environment.

Super Cool Tip™ Because quality is everyone’s problem.

Add an additional pipeline to fail builds if

quality decreases.

Two good examples arecode coverate and

linting errors.

#!/bin/bashset –eALLOWED_WARNINGS=100 #some arbitrary threshold

warnings=`grep "offenses detected" rubocop.txt | cut -d " " –f4`

if [ $warnings -gt $ALLOWED_WARNINGS ]then

echo -e "\033[31mallowed warnings $ALLOWED_WARNINGS\033[0m" echo -e "\033[31mactual warnings $warnings\033[0m" echo -e "\033[31mToo many rubocop warnings\033[0m" echo -e "\033[31mTry running 'bin/check_rubocop_against_master’\033[0m" exit 1

else echo $warnings/$ALLOWED_WARNINGS is close enough ಠ_ಠexit 0

fi

✔

✔

❌

❌

—Solomon (more or less)

“Improving quality is a lot of unglamourous

work that really adds up.”

68

Resources#keepshipping

Highly recommended talks about development, testing, and lots of interesting stuff: https://speakerdeck.com/searls

Ruby gem for parallel tests: grosser/parallel_tests

Parallel Docker testing: Jet (from Codeship)

CI/CD with Docker: http://pages.codeship.com/docker

Running commands with Docker Compose: http://docs.docker.com/compose

The perils of Docker-In-Docker: https://jpetazzo.github.io

This talk: slideshare.net/rheinwein

—Edsger Dijkstra

“Testing can be a very effective way to show the presence of bugs, but is

hopelessly inadequate for showing their absence.”

71

Testing does not guarantee that

our software works.

We test to know when our software is

definitely broken.

Work harder to know when you’re wrong.

Thank you!Laura Frank@rhein_weinlaura@codeship.com