1

2

How We Test MongoDB: Evergreen

Kyle Erf, MongoDB Inc.

3

Let’s Take a Look

44

5

6

7

Evergreen

8

We revolutionized our workflow by building our own

Continuous Integration System.

9

Our Automated Tests

Two typesUnit - run right after compileJS - large suites of integration tests tests, longer

Other tests done as part of the QA process

10

Requirements (Today)800+ hours of computer time to test one commit across all platforms

34 combinations of OSes, Architectures, and compile flags.

10 - 30 commits per day

Engineers need feedback as quickly as possible.

11

Early Automated TestingBuildbot

A few static hostsRun whenever things are available Not great for long testsHard to know exactly who broke whatComplicated Python scripts“Nightly” and “Weekly” test suites

What worked for a 5 people wasn’t working for 10+

12

Need To Do Better

Scale with loadFaster feedbackPrecise feedbackEasy to ConfigureKeep Costs Down

?

13

Write Our Own?Existing CI Tools Evergreen

No first-class cloud support Designed for elastic computing

Annoying to Configure Simple YAML Files

No simple pre-commit build support Pre-commit builds

Bad support for multiple platforms Support all of MongoDB’s platforms

Lots of work Lots of work

14

Prototype

Built on MongoDB + Go + AWS in 2013

Good Initial Results

Went all in on our own CI tool

The CI tool of our dreams

15

Features for our Use CaseDynamic, Multiplatform Host Allocation

Easily pinpoint the sources of failures

Test code before pushing

16

Dynamic Allocation

As many machines as we need

Scale with demandtask-centric rather than machine-centric

Limited by our longest test suite

Cheaper than dedicated hosts

17

How’s It Work?

18

We spin up enough hosts to try and get all scheduled tasks for a commit done within one hour.

Each host runs a statically compiled Evergreen agent, which accepts tasks from the main server.

We shut hosts down when the workload decreases.

Self-healing in case of system failures, frozen tests.

Server

Agent

Agent

New Host

19

For Multiple EnvironmentsCloud Services + Go give us access to all required

OS + Architecture combinations.

Linux, Windows, Solaris, OSX32 & 64-bit

AWS/SpotMac StadiumDigital OceanStatic Hosts

20

BenefitsOrders of magnitude speed-up

Feedback in a couple hours rather than days

We only use the resources we need

AWS Costs Per Day

21

Failure Feedback

22

Failure FeedbackEvergreen only runs a commit to a project every few hours.On failure, we run previous commits to pinpoint its source.

Like git bisect (but linear)

Commit #3 Commit #2 Commit #1

23

24

Spawn Hosts

Start up a duplicate host for more complex debugging

25

Benefits

Eliminates guesswork from failures

Less stress

Less time

26

Patch BuildsTest the most likely suites before commit

27

28

BenefitsExpected as part of any code review

Fewer platform-related failures

Faster iteration

29

Successes• More Builds on more platforms• Better, More Precise Feedback• Faster Iteration• Simpler Release Cycle• General Purpose• Looks Nice, Too!

30

Successes• More Builds on more platforms• Better, More Precise Feedback• Faster Iteration• Simpler Release Cycle• General Purpose• Looks Nice, Too!

1. Stop Commits2. Run Long Tests3. Wait :(

31

Successes• More Builds on more platforms• Better, More Precise Feedback• Faster Iteration• Simpler Release Cycle• General Purpose• Looks Nice, Too!

Build GCC

Build Machine Image

Build MongoDB

32

Successes• More Builds on more platforms• Better, More Precise Feedback• Faster Iteration• Simpler Release Cycle• General Purpose• Looks Nice, Too!

33

Worth It?

34

EvergreenSee our builds at

evergreen.mongodb.com

35

EvergreenOr try it out at

github.com/evergreen-ci/evergreenApache License - 0.9.0 Alpha

36

Questions?

Why Go?Our use of MongoDB?Technical Challenges?

How can I use Evergreen?