Continuous Integration at CartoDB

JUAN IGNACIO SNCHEZ

Software Craftsmanship Madrid

Were hiring! www.cartodb.com/jobs

First things first

YOU

http://www.cartodb.com/jobs

LETS GET TO KNOW EACH OTHER!

Who has done a software deployment, or release, this year?

Who has done a software deployment, or release, during the last 6 months?

Who has done a software deployment, or release, during the last month?

Who has done a software deployment, or release, during the last week?

Who has done a software deployment, or release, during the last day?

Tools?

Team!

Team!

Process!

(just a little)

Continuous deployment is more of a deal with

the devil than magic ;-)

me

Continuous deployment is more of a deal with the devil than magic ;-)

15 / week peak: 7

Breaking CartoDB up

Components

1. Editor

2. Platform

1. SQL API

2. Maps API

3. CartoDB.js

Products

1. SaaS

2. On-premise

3. Open Source

Product Team

1. Editor: 8

2. Data services: 5

3. Web: 6

4. APIs: 2

5. Systems: 5

The GOALS in the deal

Reduce risk

Reduce risk

SLA-own

ed

Reduce risk

impact x chance

Add value sooner

Roadmap

7 big* features /

milestone (2 weeks)

Deliver features

22 PRs each week

15 / week peak: 7

users

support

users

Summary, agile basics

(quick) feedback

(less) cost

HOW

Integration Delivery Deployment

Continuous<

Continuous Integration[CI.2] Continuous Integration is a software development

practice where members of a team integrate their work frequently, usually each person integrates at least daily - leading to multiple integrations per day. Each integration

is verified by an automated build (including test) to detect integration errors as quickly as possible.

XP (1999), Booch method (1991)

Continuous Delivery

[CD.1] software engineering approach in which teams

keep producing valuable software in short cycles and

ensure that the software can be reliably released at any time. [CD.2] You build software in such a way that the software

can be released to production at any time

Continuous Deployment

[CD.1*] software engineering approach in which teams

keep producing valuable software in short cycles and

ensure that the software can be is reliably released at any time. [CD.2*] You build software in such a way that the software

can be is released to production at any time

Continuous Deployment at CartoDB: a deal with the devil?

JUAN IGNACIO SNCHEZ

CAS 2015 Basic level talk

[CI.1]

1. Maintain a code repository

2. Automate the build

3. Make the build self-testing

4. Everyone commits to the baseline every day

5. Every commit (to baseline) should be built

6. Keep the build fast

7. Test in a clone of the production environment

8. Make it easy to get the latest deliverables

9. Everyone can see the results of the latest build

10. Automate deployment

Best practices

The CartoDB Way

[CI.1]

1. Maintain a code repository

2. Automate the build

3. Make the build self-testing

4. Everyone commits to the baseline every day

5. Every commit (to baseline) should be built

6. Keep the build fast

7. Test in a clone of the production environment

8. Make it easy to get the latest deliverables

9. Everyone can see the results of the latest build

10. Automate deployment

Best practices

Step 0: hiring

1. Initiative

2. Courage

3. Teamwork

4. Problem solving

Step 1: onboarding

1. Big picture

2. Empowerment:

deploy from the very

beginning, senior

or junior

Step 2: plan

1. Incremental

2. Iterative

3. Accept tradeoffs

Step 3: code

1. Take issue

2. Branch master

3. Code + tests

4. Staging testing

Step 4: pull request

1. Integration tests

2. Code style

3. Code review

Step 5: merge

& deploy

Step 6: evil devil

#devops

Rollbar

Kibana

No QA?

No QA team

QA Everywhere

1. Incremental, iterative

2. Small steps

3. Automated test suite

4. Code reviews

5. Dogfooding

6. Support team

7. Monitoring

8. .

() teams are often discouraged by frequent bugs quietly sneaking into their code. The common solution is to hire dedicated testers; a Quality Assurance (QA) team. This is an expensive mistake.

- Josh Steiner, Testing Rails

As your application grows, now you have to scale the number of hands on deck, who will never be as effective as automated tests at catching regressions. QA increases the time to implement features, as developers must communicate back and forth with another human. Compared to a test suite, this is costly.

- Josh Steiner, Testing Rails

This is not to say that QA is completely useless, but they should be hired in addition to a good test suite, not as a replacement. While manual testers are not as efficient as computers at finding regressions, they are much better at validating subjective qualities of your software, such as user interfaces.

- Josh Steiner, Testing Rails

The devil is in the detail

Feature flags

[FF.1] The basic idea is to have a configuration file that

defines a bunch of toggles for various features you have

pending. The running application then uses these toggles

in order to decide whether or not to show the new

feature. - Fowler

Feature flags

Feature flags

Canary Releases

[CR.1] Canary release is a technique to reduce the risk of

introducing a new software version in production by

slowly rolling out the change to a small subset of users

before rolling it out to the entire infrastructure and

making it available to everybody.

Canary Releases

Quick rollbackIntegration issues

Scaling issues

Corner cases

1

2

3

Quick rollbackDrawback

Migrations in its own PR Previous PR for preparing ORM (ActiveRecord)

Post-mortem Affected components Timing What happened

5W Actions

Smoke tests

Manual testing On-premise releases Tag

Tools

Jenkins & Travis Clinker > Jenkins

CI, deployment (Chef)

Jenkins & Travis Travis

Testing

RSpec (a little) Capybara Jasmine

Static analysis Hound / Atom

Ruby Rubocop Simplecov Reek

JS: JSHint

SCSS: SCSS-Lint

Wrapping it Up

Obstacles External Dependencies

Weak typing

Load

Integration

Costs

Riskier Have plan B

backups

rollbacks

soft deletions

Heavy instrumentalized servers

fast tests

real time alerting

Costs

Latest issues Unexpected DB locks

Integration issues

MultiJSON

Benefits Cheaper Early detection & diagnosis No release day panic Small and easy reverts.

Benefits Accelerated TTM Building the Right Product Test in actual environment Visible progress

Is Continuous Delivery suitable for you?

Is Continuous Delivery suitable for you?

\_()_/

Questions?

Thank you!

Juan Ignacio Sanchez Backend Engineer at CartoDB

juanignaciosl@cartodb.com @juanignaciosl

References

Continuous Integration [CI.1] Wikipedia: https://en.wikipedia.org/wiki/Continuous_integration

[CI.2] Fowler: http://www.martinfowler.com/articles/continuousIntegration.html

[CI.3] Duvall: Continuous Integration. Improving Software Quality and Reducing Risk http://

martinfowler.com/books/duvall.html

Continuous Delivery [CD.1] Wikipedia: https://en.wikipedia.org/wiki/Continuous_delivery

[CD.2] Fowler: http://martinfowler.com/bliki/ContinuousDelivery.html

[CD.3] Agile Alliance: http://guide.agilealliance.org/guide/cd.html

XP [XP.1] eXtreme Programming: https://en.wikipedia.org/wiki/Extreme_programming

Feature Flags [FF.1] FeatureToggle - Martin Fowler: http://martinfowler.com/bliki/FeatureToggle.html

Canary Releases [CR.1] CanaryRelease - Danilo Sato: http://martinfowler.com/bliki/CanaryRelease.html

Misc [M.1] Booch method: https://en.wikipedia.org/wiki/Booch_method

[M.2] One Hacker Way - Erik Meijer: https://vimeo.com/110554082

[M.3] https://medium.com/gutefrage-net-engineering/how-to-escape-from-the-deployment-and-

integration-hell-5c55e1628137#.o7d0n3a5h

https://medium.com/gutefrage-net-engineering/how-to-escape-from-the-deployment-and-integration-hell-5c55e1628137#.o7d0n3a5h

were hiring, take a look at www.cartodb.com/jobsAnd remember

YOU

http://www.cartodb.com/jobs