Automating Operations and Support with Salt Stack

Saurabh Surana

Agenda

• OpenStack Trove – Quick Intro• Salt Environment Overview• Automation Overview• What changes for operations with salt stack?• Salt Modules• Salt States• Challenges we are facing

OpenStack - Trove

• OpenStack’s Database as a Service Project

• REST API for creating managed database instances

• Provides mysql support out of the box

• Actively being worked on by HP, Rackspace, Mirantis….

OpenStack – Trove

TROVE API

TROVE API

TROVE TASKMANAGER

TROVE TASKMANAGER

TROVE CONDUCTOR

TROVE CONDUCTOR

DBDBRabbitMQRabbitMQ

MySQL InstanceMySQL

Instance

MySQL InstanceMySQL

Instance

OpenStack N

ovaO

penStack Nova

User 1

User 2

Salt Environment Overview

• Separate master setup per deployment

• Multi master setup

• Manage salt-master just like any other minion

Salt Environment Overview

• States and Pillar in separate repository

• States are common across all the deployments (Prod Env)

• Pillar is maintained per environment

Salt Environment Overview

• Every database instance that gets created is a minion

• Automated (but secured) way of accepting and deleting those keys

Things to consider for Operations with Salt Stack

How is it different with Salt Stack?

• Access to salt master is needed•Access to sensitive data (Pillar)•Access to infrastructure•Auditing individual user actions•Auditing Salt Command execution

Access to Sensitive data

• Correct ACLs to protect the pillar data

• Consider encrypting your pillar data

• Protecting Salt master configurations and minion keys

Access to infrastructure

• Since user is on salt master, he has access to every node in the infrastructure

• That too, as root, salt minion is running as root

• Protecting what type of actions can be performed by individual users, by controlling access to modules

Some modules worth blocking

• cmd.*• file.*• cp.*• config.*• pillar.*• ……• (There are very few which are to be considered

safe)

Auditing individual user actions

• Make sure salt commands are executed from their own user account, this can be achieved with the help of correct configurations

• Salt master logs every user action •user name, •salt command •salt minion

Auditing Salt Command Execution

• Every salt command that was executed, get recorded in salt job management framework

• Can query the details of a particular job?

• Consider using a returner, simplify serach

Controlling salt master through salt• Controlling salt master like any other salt minion

using salt framework

• We do this by installing a minion on master

• This allows use to treat the master as any other minion in the system, so we can write states, pillar and modules for it

Salt States

Salt States

• Separating states to simplify day to day operations

• Writing states to catch and possiblify mitigate user errors

Separating states

• One big state files seems simple and state forward• Try separating those states into simple group of

actions• This will allow you to run those specific actions

when they are needed and will not affect anything else

• This will be very good addition for operations

Separating states

• For example separating following from each other:– Package Installation– User creation– Configuration parameter management– System level service/package/upgrade management

• Note: Where ever required you need to maintain necessary dependencies even after separation

Separating states

• If package installation is separate, at the time of upgrading packages, users will be unaffected

• If user management is separate state file then, at the time of password rotation, nothing for packages or system level services will be affected

Writing states to catch user errors

• Writing state with necessary checks to help catch user errors

• E.g while writing states for salt-master, make sure that its getting executed on the master only. May be add a check on grains[id]

Salt Modules

• Extending salt modules

• Writing modules to catch user errors

Extending Salt Modules

• Consider extending salt modules:•If existing salt module is possible to cause harms

if misused•If for certain action user needs to run multiple

salt commands•If completing an action, user will need values

from pillar data

Thanks • Demo Scripts/files @

https://github.com/saurabhsurana/salt-stack-demo