spot trading - a case study in continuous delivery for mission critical financial services

Automated deployments for mission-critical financial services

Jeremy Alons Systems Engineer, Spot Trading LLC Views and statements are my own and may not reflect those of my employer

v.8

A case study in continuous delivery

Why?

• Previously, deployments were time consuming

• Took hours of developer time, sometimes spread over weeks

• Manual process meant many problems at many points

Our goals• Rapid feedback

• Tie configuration to deployment

• Be highly parallelized, if possible

• Infrastructure as code

• Repeatable

Challenges

• Lots of unique software

• Differences force complexity

• Misused the tools we had originally

• Required additional tooling (that continues today)

Steps: Build

Deploy Config

Run

Steps: Build

Deploy!Config

Run

Steps: Build

Deploy Config!

Run

Build: Bamboo (or Jenkins, travis-ci, etc) executes commands

i.e. gcc -o foo foo.c sh tests.sh

etc

Source code in. Artifacts out.

Now what? We need to get artifacts to where they need to go

This poses problems. • What hosts? • How do we get correct versions to those hosts? • Are we sure those hosts are suitable for the application?

SaltStack to the rescue!

Fast, scalable and flexible systems management software for data center

automation, cloud orchestration, server provisioning, configuration management

and more

- www.saltstack.com

http://www.saltstack.com

Salt is extremely extensible• External pillars

* Just python

• Interface with ZooKeeper for host discovery

• ZooKeeper maintains source of truth

• But how does Bamboo talk to ZooKeeper?

• Facilitate the communication through Salt

Grains

• Static information about systems

• Loaded at minion start time

• OS (major, minor), ipv[4|6], etc

• Something you ask the server

Interface to derive information about the underlying system

Pillar

• Something you tell a server

• “pillar” is a dictionary of items sent to minions

• ext_pillar are derived by the master by executing python

• Still sent to minions

Pillar is an interface for Salt designed to offer global values that can be distributed to all minions.

Grain: Ask Pillar: Tell

server1server2

Salt Master

ZooKeeper

Stash

Bamboo

Zoom

Build triggered

Selects build to deploy

Tells salt to find

matching daemons

Returns results

Runs deploy state

Asks forhosts

Return hosts to target

Displays real timestatus and

dependency map

Developer

Pushes commit1

23

4

5

6

7

8

0

Bamboo talks to salt via Salt’s RESTful api

Salt talks to ZooKeeper via an external pillar

Salt asks ZooKeeper “What hosts are currently responsible for app Foo?”

ZooKeeper responds, we run a series of state files on the targeted servers

States• Check stats (memory, disk, etc)

• - cmd.run salt://deploy/check_host.py

• Apply system settings (buffers, os monitoring, etc)

• Install software

• yum install Foo-release-2.3-10

After build, package into rpm

• Use the tools available

• Versioning

• Well understood

• Reporting baked in

RPM

• Specify version

• Allows control over latest build, n-1, etc

• Passed through via pillar at install time, fed by Bamboo deployment plan

• Allows Bamboo to drive version released

API wrapper

• Home grown deployment software

• Command line utility that converts bamboo variables to salt states

• Deploys itself via the deployment process - Dog fooding the deployment process

deploy state# Run the installation on the targeted server deploy_{{ application_name }}: pkg.installed: - name: {{ application_name }} - version: {{ pillar[‘deploy_pillar’][‘version’] }} - refresh: True - enablerepo: spot_applications - require: - sls: spot_applications_repo

Things we’ve learned• In the end, it’s all just bits on disk

• All software is just software, and (should be) equal

• Be specific and exact

• Catch issues soon (test early, test often)

• Being clever will come back to haunt you (after all, this is just software development)

Highlights

• Implemented deployment pipeline

• Artifact of the build

• Bringing components together

• Repeatable, rapid feedback

Thank you.

Questions?

email: [email protected]

For more about some of the interesting work Spot is doing, check our Engineering Blog at

http://www.spottradingllc.com/technology/

mailto:[email protected]

http://www.spottradingllc.com/technology/

spot trading - a case study in continuous delivery for mission critical financial services

Technology