Free Classifiedswww.olx.com

Buenos Aires, April 2014

Continuous DeliveryRelease Management

2

Agenda

1. Initial Situation2. Initial Setup3. Continuous Delivery Adoption - How we started4. Deployment stages5. Application configuration deployment6. Introducing Octopush7. Architecture evolution8. Tooling9. Culture changing10. Lessons learned11. What we have achieved12. Next steps13. Q&A

3

4

Initial Situation

Situation

Complication

BigBang/Stressful releases, once every two weeksMultiple Development Teams -> # deployments increasing Few automation - Most of the tasks were manualMonolithic platform / Dependency hellManual ConfigurationSVNSome teams doing CI Distrust & escepticism

• Business Risk Product Delivery Product Quality Need to shorten the time to market when adding new

features Competitive advantage: Need more efficient processes and

reduce operational costs.

• Growing teams• Going Global / distributed team / different time zones

5

SVN

CI Server

RM Server

ssh/console

Trigger job

Dev Team I

Release Management

Initial CI and RM Setup

Dev Team II

6

Initial Picture

7

Decision

8

Initial Assessment

Source: http://www.infoq.com/articles/Continuous-Delivery-Maturity-Model

9

• Sharing the situation and necessity• Management commitment• Resources allocated

• Reduce Errors / Lower Stress / Lower Delta Changes Release

• On Demand Deployment/ Built Quality In

• Everybody Is Responsible for the Delivery Process/ Continuous

Improvement

• Establish Phases and goals

• Scope by component

• Assumptions / expectations

• Project Risk Identification

• Continuous Integration• Continuous Delivery• Continuous deployment

Commitment

Concepts alignment

Setting Goals

Project Management

Continuous Delivery Adoption

10

Integration/Acceptance EnvDev Env

High Level View

CommitStage

DeploymentStage

AcceptanceStage Live

Manual gate :: RM Button

Production Env

Automatic :: Web-hook

11

Integration Jenkins

Commit Stage

Git Master/Branch Artifactory Release Jenkins

Checkout

Publish binaries

Trigger Release

Checkpoint:- build or pre-compile- run unit test & coverage

from Git master branch to Artifactory

Deployment Stage

[ENV available?]::Go::Wait

Commit hookCommit

Build

12

Release Jenkins

Deployment Stage

from Integration Jenkins

Git repoArtifactoryAcceptance Env.

[If code] Get Binaries

[If config] Get Config

from Artifactory to Acceptance Env

To Integration Jenkins

Run Deploy Script

Run smoke test

Automatic Rollback upon

failure

Code OR Config,

NEVER both

13

Integration/Dev Jenkins

Acceptance Stage

from Release Jenkins

Acceptance Env.

from Acceptance Env to Production Gate

To Release Jenkins

Run Acceptance Tests

Run Performance Tests*

* Using YSlow for frontend testing

Automatic Rollback upon

failure

14

Application configuration deployment

Config files, from the repo to the environment

git-cryptgit-crypt

15

Architecture evolutionFrequency/Complexity/Impact/Risk/Speed deployment

16

Introducing Octopush

Jenkins Mobile

Jenkins Frontend

Jenkins Backend

Jenkins blah

Make deployment request

Deploy

Enqueues jobPick next available jobCall Jenkins to deployPolls/notify on completion

Orchestrate Jenkins Hell!

Open SourceOpen Source

Call deployment

job

17

Pipeline layers

18

The Revolution will not be centralized

19

Remaining servers

Alert

ROLLBACK

WaitMonitorCheck

Canary ReleaseRelease pool

20

Tooling

REST APIREST API

22

Culture Changing

23

Architecture

Smoke Test

Infrastructure-as-code

Conventions

• Application architecture.• Tooling architecture should also be layered and Service oriented.• Integrate your own tools with Jenkins JIRA, IRC, Graphite, most of them

have a REST API service. No silver bullet• Divide configuration

• Health checks!

• Track & version your scripts, follow Software design principles• Same scripts/infra everywhere!, use config files to resolve hostname

issues• Scripts should be idempotent

• Apps/repos/users/package naming.

Canary Releases• Use a Testing pool, automate!.

Lessons learned 1/2

24

Communication• Clear channel• IRC• Automate.

Strategy

• MVP• Advertise progress/delays, achievements, use charts/hard data. • Early adopters.

Lessons learned 2/2

25

• Tripled our Production deployment throughput .

• More confident and error-free releases.

• Less stress, near-zero manual work, easier, faster deploys

ConclusionsReleases

Achievements

Octopush decentraliza

tion

26

Next steps

. 1.Native packages

1.Deploy integration with Puppet

1.Virtualization / Docker

27

Q&A

.

28

Thank You (We are hiring DEVOPS)

Continuous Delivery TeamDiego Garber @diegarberGoogle+ Communities: OLX Engineering