intelligent infrastructure with saltstack

Intelligent Infrastructure with SaltStack

Love Billingskog Nyberg Infrastructure and Opera=ons Team Lead

Event > React > Orchestrate

Hi!

Love @jacksoncage

bloglovintens of millions of monthly unique visitors hundred of millions requests per day couple of million stories parsed per day

hybrid bare-metal servers/cloud vm’s 4 salt masters ~90 salt minions

salt @ bloglovinParts we use!

States/Pillars/Grains Pillarstack* Orchestrate Reactor Beacons Engines

Parallel execu=on to run all the resources concurrently

Event driven to monitor and react dynamically only to changes

Next genera=on

Next genera=on

Parallel execu=on

root@salt-dev:~# salt '*' test.ping web101: True db101: True ship101: True master: True

Easy to target remote execu5on against mul5ple targets

Parallel execu=onRun commands via execu5on modules

root@salt-dev:~# salt web* service.stop nginx web101: True

Tradi=onal configura=onSe;ng state on interval or on run

Event driven configura=onSe;ng state when needed or change has happened

GoalsMonitoring - How can salt react to monitoring events and adjust

system without any manual interference

Minions - When a new device is connected get it’s data and no=fy me, chat Vw!

Containers - When a new container starts, update load balancer for that cluster and start sending traffic. Also when a container stops, get logs and restart and send logs to team

Eventssalt/job/20160413070418586332/new { "_stamp": "2016-04-13T11:04:18.587129", "arg": [], "fun": "test.ping", "jid": "20160413070418586332", "minions": [ "web101" ], "tgt": "web101", "tgt_type": "glob", "user": "sudo_love" }

salt-run state.event pretty=True

Eventssalt/job/20160413070418586332/ret/web101 { "_stamp": "2016-04-13T11:04:18.789831", "cmd": "_return", "fun": "test.ping", "fun_args": [], "id": "web101", "jid": "20160413070418586332", "retcode": 0, "return": true, "success": true }

State eventsEnabling state events causes the Salt minion to send progress events as

each func5on in a state run completes execu5on.

state_events: True

minion

State eventsWhich can become an over load, let’s fire only on specific states

installed/vim: pkg.installed: - name: vim - fire_event: installed/vim

minion

Custom eventEnabling state events causes the Salt minion to send progress events as

each func5on in a state run completes execu5on.salt-call event.send /deploy/test/case '{"data": "Ready"}'

/deploy/test/case { "_stamp": "2016-04-13T11:01:24.553140", "cmd": "_minion_event", "data": { "__pub_fun": "event.send", "__pub_jid": "20160413070124500641", "__pub_pid": 17046, "__pub_tgt": "salt-call", "data": "Ready" …

minion

Beacons

beacons: service: nginx: onchangeonly: True

Hook into system processes. Translate external events into the salt event bus

minion

Engines

engines: docker_events: docker_url: unix://var/run/docker.sock

Salt Engines are long-running, external system processes that leverage Salt.

minion

Great! Events for everything!

ReactorSalt’s Reactor system gives you the ability to trigger ac5ons in response to any event.

reactor: # Master config section "reactor"

- 'salt/minion/*/start': # Match tag "salt/minion/*/start" - /srv/reactor/start.sls # Things to do when a minion starts

- 'deploy/containers': # Match tag "deploy/containers" - /srv/reactor/deploy-containers.sls # Things to do when a container is deployed

- 'salt/netapi/hook/api/devlovin/create': # Listing on salt api for /devlovin/create - /srv/reactor/devlovin-create.sls # Things to do when a create request is sent

master

ReactorSalt’s Reactor system gives you the ability to trigger ac5ons in response to any event.

Remote execu=on: run an execu=on module on the targeted minions.

Runners: These are tasks you would start using salt-run.

Wheel: Wheel commands manage your Salt environment.

master

ReactorSalt’s Reactor system gives you the ability to trigger ac5ons in response to any event.

<section id>: local.<function>: - tgt: <target> - arg: <arguments>

master

# This script is to be called from the reactor system {%- set application = data['data']['application'] %} {%- set container = data['data']['container'] %} {%- set target = data['data']['target'] %} {%- set tag = data['data']['tag'] %}

sync-states: local.saltutil.sync_states: - tgt: {{ target }} deploy-containers: local.state.sls: - tgt: {{ target }} - arg: - containers.{{ container }} - kwarg: pillar: containers: imagetag: {{ application }}: {{ tag }}

master

highstate_run: local.state.highstate: - tgt: db101

call_some_orchestrate_file: runner.state.orchestrate: - mods: some_orchestrate_file - pillar: stuff: things

Highstate run

Star5ng a orchestrator runner

master

OrchestrateThe orchestrate runner generalizes the Salt state system to a Salt master context.

master

# Create AWS EC2 instance create-instance: salt.state: - tgt: {{ master }} - sls: devlovin.cloud.create - concurrent: True - pillar: devlovinreact: instancename: {{ basename }}{{ username }}

# Run highstate highstate-run: salt.state: - tgt: {{ basename }}{{ username }} - highstate: True - pillar: devlovinreact: username: {{ username }} - require: - salt: create-instance

master

Demo

Monitoring - How can salt react to monitoring events and adjust system without any manual interference

Demo

Minions - When a new device is connected get it’s data and no=fy me, chat Vw!

Demo

Containers - When a new container starts, update load balancer for that cluster and start sending traffic. Also when a container stops, get logs and restart and send logs to team

Ques=ons?

Love Billingskog Nyberg @jacksoncage

love@bloglovin.com

Thanks!

Love Billingskog Nyberg @jacksoncage

love@bloglovin.com

Use the SaltConf16 event app to provide feedback