real-time cloud management with saltstack

Real-time cloud management with Salt

Unlocked.io Seth House <shouse@saltstack.com>

2013-06-14

1 / 47

Salt philosophy

1 Salt philosophyBuilding blocksWhat are our building blocks?Building blocks and complexitySuccess comes from simplicitySalt is aptly namedFlow + state = EVERYTHINGBuilding on state and flowBuilding blocks for infraBuilding remote exBuilding statesBuilding cloudBuilding monitoringConclusion

2 Salt internals (briefly)

3 Salt speed

4 Minion data

5 Events

6 Schedules

7 What’s coming

2 / 47

Salt philosophy Building blocks

Building blocks

Figure: By Stilfehler (Own work) GFDL or CC-BY-SA-3.0-2.5-2.0-1.0

3 / 47

Salt philosophy Building blocks

Building blocksEverything we have is built on earlier generationsBuilding blocks of history

AgricultureCitiesGovernmentsMathematicsManufacturingLanguage

4 / 47

Salt philosophy What are our building blocks?

What are our building blocks?Computers Systems

Data centers are the modern platform of inventionLarge scale computing drives virtually all high level human interactions

Data centers themselves are becoming the building blocks

5 / 47

Salt philosophy Building blocks and complexity

Building blocks and complexityAs systems grow they become more complex

Increased complexity makes using larger building blocks impossibleBuilding blocks used today are simple

Easy to useEasy to understandYet powerful

6 / 47

Salt philosophy Success comes from simplicity

Success comes from simplicityViable building blocks are fundamentally simple

Things that are in themselves simple to understand and use

Concepts that can cross barriers and cleanly apply to many higher leveluses

YAML is a great example

Unix is a great example

7 / 47

Salt philosophy Salt is aptly named

Salt is aptly named

Figure: By Alan Chia (Lego Color Bricks) CC-BY-SA-2.0

8 / 47

Salt philosophy Salt is aptly named

Salt is aptly namedThe name is the vision

Salt is in nature, a building block

Salt is in cooking, a building block

Salt is in life, a building block

Yes, this is where the logo came from

Yes, this is why it is Salt STACK

9 / 47

Salt philosophy Flow + state = EVERYTHING

Flow + state = EVERYTHINGCore idea behind Salt

Flow = Access and control

State = The functional state of the system

Between them All things are controlled

10 / 47

Salt philosophy Building on state and flow

Building on state and flowBasics

Remote execution built on FlowConfig Management built on State

Ramping UpMonitoring build on Flow + StateCloud management is Flow + StateAuto healing is Flow + FlowOrchestration is Flow + StateDistributed Deployment is Flow + State

11 / 47

Salt philosophy Building blocks for infra

Building blocks for infraSalt has “Configuration Management”

Salt has “Remote Execution”

12 / 47

Salt philosophy Building remote ex

Building remote exAsync Message Bus

Micro publish commands

Minion Side Logic

Return to arbitrary location

13 / 47

Salt philosophy Building states

Building statesAll about data

Low level data structure calls state functions

High level data structure defines state calls

Languages are now arbitrary

14 / 47

Salt philosophy Building cloud

Building cloudRemote Ex controls hypers

Remote ex commands hypers

All runs on Live data

15 / 47

Salt philosophy Building monitoring

Building monitoringRemote Ex build on system libs

Data gathering is already there!

States used to detect the state of a system

States can declare that the state of a system is out of bounds

Fires back to the event bus

16 / 47

Salt philosophy Conclusion

Conclusion

Figure: By Ex13 (Own work) CC-BY-SA-3.0

17 / 47

Salt philosophy Conclusion

ConclusionSalt is comprised of many building blocks

The underlying components are available, but they do not need to beinterfaced with

Salt is made to morph into the needs of the infra

Made to make small infrastructures easy, and large infrastructurespowerful

18 / 47

Salt internals (briefly)

1 Salt philosophy

2 Salt internals (briefly)MasterMinionsExecution modulesExecution exampleState modulesState module example

3 Salt speed

4 Minion data

5 Events

6 Schedules

7 What’s coming

19 / 47

Salt internals (briefly) Master

Mastersalt-master -d

Open two ports (pub/sub & reply channel)

20 / 47

Salt internals (briefly) Minions

Minionssalt-minion -d

Connect to the master

No open ports required

Listens for pubs from the master

/etc/salt/minion:#master: salt

21 / 47

Salt internals (briefly) Execution modules

Execution modulesContain all the functionality

22 / 47

Salt internals (briefly) Execution example

Execution examplesalt ’web-

*

’ network.interfaces

23 / 47

Salt internals (briefly) State modules

State modulesWrap execution modules

Before-check

test=true

Call out to execution modules

After-check

24 / 47

Salt internals (briefly) State module example

State module exampletop.sls:

base:

’web-

*

’:

- httpd

httpd.sls:httpd:

pkg:

- installed

25 / 47

Salt speed

1 Salt philosophy

2 Salt internals (briefly)

3 Salt speedCommunicationpub/sub

4 Minion data

5 Events

6 Schedules

7 What’s coming26 / 47

Salt speed Communication

CommunicationZeroMQ

msgpack

27 / 47

Salt speed pub/sub

pub/subAsynchronous

Minions determine targeting match

Minions do all the work

28 / 47

Minion data

1 Salt philosophy

2 Salt internals (briefly)

3 Salt speed

4 Minion dataPeerPeer exampleSalt MineSalt Mine exampleReturnersReturner full-circle example

5 Events

6 Schedules

7 What’s coming

29 / 47

Minion data

Live data; peer interface

Recent data; Salt mine

Historical data; returners

30 / 47

Minion data Peer

Peer/etc/salt/master:

peer:

lb-.

*

:

- network.interfaces

Be mindful of data security

Communication still goes through the master

31 / 47

Minion data Peer example

Peer exampleConfiguring haproxy.cfg:

{% for server,ip in

salt[’publish.publish’](

’web

*

’,

’network.interfaces’,

[’eth0’]).items() %}

server {{ server }} {{ ip[0] }}:80 check

{% endfor %}

32 / 47

Minion data Salt Mine

Salt Mine/etc/salt/{master,minion}:

mine_functions:

network.interfaces: [eth0]

mine_interval: 60

New in Salt v0.15

Either master or minion config

Be mindful of data security

33 / 47

Minion data Salt Mine example

Salt Mine exampleConfiguring haproxy.cfg:

{% for server,ip in

salt[’mine.get’](

’web-

*

’,

’network.interfaces’,

[’eth0’]).items() %}

server {{ server }} {{ ip[0] }}:80 check

{% endfor %}

34 / 47

Minion data Returners

Returners/etc/salt/{master,minion}:

redis.db: 0

redis.host: myredis

redis.port: 6379

Minions write directly

Can be read into Pillar via ext_pillar

35 / 47

Minion data Returner full-circle example

Returner full-circle exampleCollect the data:

salt ’web-

*

’ network.interfaces eth0 \

--return redis_return

Fetch the data via a custom ext_pillar module.Use the data:

{% for server,ip in

salt[’pillar.get’](’web.ip_addrs’, {}).items() %}

server {{ server }} {{ ip[0] }}:80 check

{% endfor %}

36 / 47

Events

1 Salt philosophy

2 Salt internals (briefly)

3 Salt speed

4 Minion data

5 EventsFire eventsWatch for events (manually)Reactor (react to events)

6 Schedules

7 What’s coming 37 / 47

Events Fire events

Fire eventssalt ’lb-

*

’ event.fire_master \

refresh_pool loadbalancer

38 / 47

Events Watch for events (manually)

Watch for events (manually)Some assembly required

salt/tests/eventlisten.py

Coming soon to salt-api

39 / 47

Events Reactor (react to events)

Reactor (react to events)/etc/salt/master:

reactor:

- loadbalancer:

- /src/reactor/refresh_pool.sls

/src/reactor/refresh_pool.sls:{% if data[’type’] == ’refresh_pool’ %}

highstate_run:

cmd.state.highstate:

- tgt: lb-

*

{% endif %}

40 / 47

Schedules

1 Salt philosophy

2 Salt internals (briefly)

3 Salt speed

4 Minion data

5 Events

6 SchedulesAdd eventsStats gathering

7 What’s coming41 / 47

Schedules Add events

Add events/etc/salt/{master,minion} (or pillar):

schedule:

highstate:

function: state.highstate

minutes: 60

42 / 47

Schedules Stats gathering

Stats gatheringschedule:

uptime:

function: status.uptime

seconds: 60

returner: redis

meminfo:

function: status.meminfo

minutes: 5

returner: redis

43 / 47

What’s coming

1 Salt philosophy

2 Salt internals (briefly)

3 Salt speed

4 Minion data

5 Events

6 Schedules

7 What’s comingMonitoring statesData resolution

44 / 47

What’s coming

Salt v.0.next

45 / 47

What’s coming Monitoring states

Monitoring statesConfigure inline with existing states

Individual components are in place

Needed: glue

Needed: alerting

46 / 47

What’s coming Data resolution

Data resolutionTime-series data

Thin and/or summarize older and older data

Free with some returners

47 / 47