rapid, reproducible builds

R A P I D

REPRODUCIBLE BUILDStreating everything like production

Dewald Viljoen @dewald_v

who am i?

2

HOW'S THIS TALK GOING TO WORK

and what are we actually talking about?

3

"a server is a server is a server, there really are no test servers, build

servers, dev servers and production servers... and spoons of course"

- @dewald_v

4

DOCKER IN 5 MINUTESa quick and dirty look into images, containers and all that lies below

5

DOCKER IN 5 MINUTES - IMAGES

6

NGINX IMAGE

UBUNTU 14.04

ADD NGINX

ADD NGINX CONFIGLayer 3

Layer 2

Base (Layer 1)

DOCKER IN 5 MINUTES - CONTAINERS

7

NGINX CONTAINER

UBUNTU 14.04

ADD NGINX

ADD NGINX CONFIGLayer 3

Layer 2

Base (Layer 1)

CONTAINER R/W FILESYSTEMLayer 4 (Only in the Container)

NGINX-1

CONTAINER R/W

NGINX-2

CONTAINER R/W

NGINX-3

CONTAINER R/W

NGINX IMAGE

DOCKER IN 5 MINUTES - CONTAINERS AND LAYERS

8

DOCKER IN 5 MINUTES - CONTAINER VS VM

9

Image credit: Docker Inc.

what can we use containers for?

10

WORKSTATIONS AND BUILD AGENTSthe darkest corners of the mind

11

what's on a workstation?

12

A TYPICAL WORKSTATION

13

WORKSTATION

A TYPICAL WORKSTATION

14

WORKSTATION

Basically everything and anything you would

need to work on your stack

what's on a build agent?

15

A TYPICAL BUILD AGENT

16

(SUPER) BUILD AGENT

A TYPICAL BUILD AGENT

17

(SUPER) BUILD AGENT

Basically everything and anything you would need to build your app

what's the alternative?

18

DOCKER-POWERED BUILDSapplying images and containers to build pipelines

19

NODEJS BUILD IMAGE

THE BUILD IMAGE - NODE + GRUNT

20

UBUNTU 14.04

ADD NODE 4.2.4

ADD GRUNT

ADD PACKAGE.JSON & NPM INSTALL

Base Image

Build Runtime

Build Config & Dependencies

Build Tool

MAVEN BUILD IMAGE

THE BUILD IMAGE - JAVA + MAVEN

21

UBUNTU 14.04

ADD JDK 1.8

ADD MAVEN 3.3.3

ADD MAVEN SETTINGS.XML

Base Image

Build Runtime

Build Config

Build Tool

THE BUILD CONTAINER

22

MAVEN BUILD IMAGE

+

=

SOURCE CODE

BUILD ARTIFACT

JAVA-APP-BUILD-1

CONTAINER R/W

JAVA-APP-BUILD-3

CONTAINER R/W

JAVA-APP-BUILD-2

CONTAINER R/W

THE BUILD CONTAINER - MULTIPLIED

23

MAVEN BUILD IMAGE

ENVIRONMENTS, ENVIRONMENTS, ENVIRONMENTS...

24

LOCAL MACHINE BUILD AGENT

DOCKER REGISTRY

THE BUILD CONTAINER - EVEN MORE MULTIPLIED

25

MAVEN 3 MAVEN 2 NODEJS RUBY

JAVA-APP-BUILD

JAVA-APP-BUILD

NODE-APP-BUILD

RUBY-APP-BUILD

BUILD AGENT

ARTIFACT ARTIFACT ARTIFACT ARTIFACT

OF PIPELINES AND CONTAINERShow do we use this then?

26

enterprises build all the things...

27

THE SUPER BUILD AGENT

28

BUILD AGENT• Complex

• Can be containerised but needs a lot of config

• Holds all build tools and configuration for any type

of build it can service

• Results in high build-contention

• Difficult to scale

• Pushes artifacts to repositories

MAVEN 3

MAVEN 2

JDK 1.5

JDK 1.6

JDK 1.7

RUBY

PYTHON

GRADLE

THE MICRO BUILD AGENT

29

BUILD AGENT• Very simple

• Can be easily containerised

• Holds build tools and configuration for only the

type of build it can service

• Easier to scale

• Can be managed with configuration as code

• Pushes artifacts to repositories

MAVEN 3

JDK 1.7

THE CONTAINER BUILD AGENT

30

BUILD AGENT

DOCKER

• Very simple

• Can be containerised it self (Docker in Docker)

• Doesn't have any build dependencies inside, only

docker

• Pulls build containers from a registry

• Simple to scale

• Pushes artifacts to repositories

• Basically a micro agent running only Docker

so what does a containerised build pipeline look like?

31

PIPELINING CONTAINERS

32

Application build directly on agent

Failed due to environmental issues

PIPELINING CONTAINERS

33

Build Container (Maven & JDK on Ubuntu)

Application build in container

PIPELINING CONTAINERS

34

Build Container (Maven & JDK on Ubuntu)

Application Builds in containers

THE GO SCRIPTnot configuring the build on the server

35

what happens when the build server gets killed?

36

BUILD CONFIGURATION

37

Backup your configuration

Configuration as code

Keep as little configuration on the server as possible

Keep build configuration next to the app it builds

THE GO SCRIPT

38

A Script for your build actions

Knows the commands to build your app

Keep it simple. Bash is a good start.

Everything that works on the build server should work UNMODIFIED on the local environment.

Build server should just call this script

Use environment variables for environment changes

PUTTING IT ALL TOGETHERdemo time

39

live demo time... (cross your fingers)

40

DEWALD VILJOEN @dewald_v

dewaldv@thoughtworks.com

THANKS! QUESTIONS?