intelligent infrastructure with saltstack
TRANSCRIPT
Intelligent Infrastructure with SaltStack
Love Billingskog Nyberg Infrastructure and Opera=ons Team Lead
Event > React > Orchestrate
Hi!
Love @jacksoncage
bloglovintens of millions of monthly unique visitors hundred of millions requests per day couple of million stories parsed per day
hybrid bare-metal servers/cloud vm’s 4 salt masters ~90 salt minions
salt @ bloglovinParts we use!
States/Pillars/Grains Pillarstack* Orchestrate Reactor Beacons Engines
Parallel execu=on to run all the resources concurrently
Event driven to monitor and react dynamically only to changes
Next genera=on
Next genera=on
Parallel execu=on
root@salt-dev:~# salt '*' test.ping web101: True db101: True ship101: True master: True
Easy to target remote execu5on against mul5ple targets
Parallel execu=onRun commands via execu5on modules
root@salt-dev:~# salt web* service.stop nginx web101: True
Tradi=onal configura=onSe;ng state on interval or on run
Event driven configura=onSe;ng state when needed or change has happened
GoalsMonitoring - How can salt react to monitoring events and adjust
system without any manual interference
Minions - When a new device is connected get it’s data and no=fy me, chat Vw!
Containers - When a new container starts, update load balancer for that cluster and start sending traffic. Also when a container stops, get logs and restart and send logs to team
Eventssalt/job/20160413070418586332/new { "_stamp": "2016-04-13T11:04:18.587129", "arg": [], "fun": "test.ping", "jid": "20160413070418586332", "minions": [ "web101" ], "tgt": "web101", "tgt_type": "glob", "user": "sudo_love" }
salt-run state.event pretty=True
Eventssalt/job/20160413070418586332/ret/web101 { "_stamp": "2016-04-13T11:04:18.789831", "cmd": "_return", "fun": "test.ping", "fun_args": [], "id": "web101", "jid": "20160413070418586332", "retcode": 0, "return": true, "success": true }
State eventsEnabling state events causes the Salt minion to send progress events as
each func5on in a state run completes execu5on.
state_events: True
minion
State eventsWhich can become an over load, let’s fire only on specific states
installed/vim: pkg.installed: - name: vim - fire_event: installed/vim
minion
Custom eventEnabling state events causes the Salt minion to send progress events as
each func5on in a state run completes execu5on.salt-call event.send /deploy/test/case '{"data": "Ready"}'
/deploy/test/case { "_stamp": "2016-04-13T11:01:24.553140", "cmd": "_minion_event", "data": { "__pub_fun": "event.send", "__pub_jid": "20160413070124500641", "__pub_pid": 17046, "__pub_tgt": "salt-call", "data": "Ready" …
minion
Beacons
beacons: service: nginx: onchangeonly: True
Hook into system processes. Translate external events into the salt event bus
minion
Engines
engines: docker_events: docker_url: unix://var/run/docker.sock
Salt Engines are long-running, external system processes that leverage Salt.
minion
Great! Events for everything!
ReactorSalt’s Reactor system gives you the ability to trigger ac5ons in response to any event.
reactor: # Master config section "reactor"
- 'salt/minion/*/start': # Match tag "salt/minion/*/start" - /srv/reactor/start.sls # Things to do when a minion starts
- 'deploy/containers': # Match tag "deploy/containers" - /srv/reactor/deploy-containers.sls # Things to do when a container is deployed
- 'salt/netapi/hook/api/devlovin/create': # Listing on salt api for /devlovin/create - /srv/reactor/devlovin-create.sls # Things to do when a create request is sent
master
ReactorSalt’s Reactor system gives you the ability to trigger ac5ons in response to any event.
Remote execu=on: run an execu=on module on the targeted minions.
Runners: These are tasks you would start using salt-run.
Wheel: Wheel commands manage your Salt environment.
master
ReactorSalt’s Reactor system gives you the ability to trigger ac5ons in response to any event.
<section id>: local.<function>: - tgt: <target> - arg: <arguments>
master
# This script is to be called from the reactor system {%- set application = data['data']['application'] %} {%- set container = data['data']['container'] %} {%- set target = data['data']['target'] %} {%- set tag = data['data']['tag'] %}
sync-states: local.saltutil.sync_states: - tgt: {{ target }} deploy-containers: local.state.sls: - tgt: {{ target }} - arg: - containers.{{ container }} - kwarg: pillar: containers: imagetag: {{ application }}: {{ tag }}
master
highstate_run: local.state.highstate: - tgt: db101
call_some_orchestrate_file: runner.state.orchestrate: - mods: some_orchestrate_file - pillar: stuff: things
Highstate run
Star5ng a orchestrator runner
master
OrchestrateThe orchestrate runner generalizes the Salt state system to a Salt master context.
master
# Create AWS EC2 instance create-instance: salt.state: - tgt: {{ master }} - sls: devlovin.cloud.create - concurrent: True - pillar: devlovinreact: instancename: {{ basename }}{{ username }}
# Run highstate highstate-run: salt.state: - tgt: {{ basename }}{{ username }} - highstate: True - pillar: devlovinreact: username: {{ username }} - require: - salt: create-instance
master
Demo
Monitoring - How can salt react to monitoring events and adjust system without any manual interference
Demo
Minions - When a new device is connected get it’s data and no=fy me, chat Vw!
Demo
Containers - When a new container starts, update load balancer for that cluster and start sending traffic. Also when a container stops, get logs and restart and send logs to team
Thanks!
Love Billingskog Nyberg @jacksoncage
Use the SaltConf16 event app to provide feedback