Михаил Щербаков - Нестандартное использование puppet в...

Non-traditionaluse of Puppet

Mike ScherbakovMoscow, September 2013

1. Initial conditions2. Push approach with MCollective3. Data Flow4. Next steps

Agenda

● Heterogenous servers● User-defined roles for distributed application● Operations

○ Remove / add nodes○ Add roles and deploy

● Deployment is activated by user● Multiple envs, different versions of code to be

deployed● Simultaneous deployments and other actions

1.1 Use case

1. Puppet Master serving the configuration code2. Puppet agents are ran by cron every 30min 3. Text file site.pp with definition what to deploy

and where4. PuppetDB to share data between nodes5. ENC to get variables from external source6. Hiera: Key-value store

1.2 Traditional Puppet Usage

1. Use Puppet2. It should run by click on Deploy, not every 30min3. Multiple environments4. Simultaneous deployment where possible5. Operations (redeployment, adding roles, etc.)

1.3 Our Initial Requirements

1.4 Our use case & Puppet

● Publish-subscribe middleware over AMQP (STOMP)

● Simple agents, written in Ruby● Agents on nodes can be run by simple command

from master node● Agent to run puppet already existed● MCollective can be used as a library

2.1 Marionette Collective

2.2 We thought

● Multithreading issues● Existing agent for puppet had multiple issues● Time synchronization sensitive● No reconnect after network failure

2.3 And we tried...

2.4 Finally it worked out

3.1 Data Flow

UserInput

Hardcoded

HardwareDiscoveredData

Data Calculation Puppet

Expected:require ‘puppet’

Puppet::Resource::Catalog.compile(all-data)

Puppet.apply

Actually got:

….monkey-patching of Puppet code, which worked only for single Puppet release.POC at http://goo.gl/xdcjlL

3.2 Not so ideal with Puppet...

Pros: unknown for our purposes

Cons:

● site.pp is code, and logic to write it properly is required

● concurrency issues if simultaneous deployment

3.3 site.pp

Data Calculation

Puppet agent

site.pp Puppet master

Pros:

● Simple well-known mechanism of passing params

Cons:

● Architecture gets complicated○ Have to make sure that nothing changes data

for ENC before puppet reads○ Hard to develop: data must be in storage

3.4 ENC

Data Calculation

Puppet agent

Puppet master

Storage

EN

C

Cons:

● Additional point of failure● Two storages: need to handle consistency● Additional dependency: Cobbler is not just PXE

anymore, hard to replace

3.5 ENC + Cobbler YAML Storage

Data Calculation

Puppet agent

Puppet master

CobblerYAML Storage

EN

C

Pros:

● Puppet Master is optional● Easy to debug and develop: all data in one file

Cons:

● Facts can be strings only● All variables loaded are global

3.6 Passing parameters via facts

Data Calculation

Puppet agent

factsonnodes

Puppet master

● YAML config is generated for every puppet run● Placed to /etc/ by MCollective

3.6.1 YAML config is created on node

Data Calculation

--- role: computenova: - rabbitmq: \":5672\" api: \":8774\"ip: 10.20.0.2/24

data = YAML.load_file(‘/etc/naily.facts`)

Facter.add(‘fuel_settings_yaml`) do

set_code { data.to_yaml }

end

3.6.2 Puppet reads YAML into fact

$fuel_settings = parseyaml($fuel_settings_yaml)

case $::fuel_settings[‘role’] {

“compute” : {

include common::compute

…

3.6.3 Parse your fact in site.pp

3.7 And we got it running

Main reasons:

1. Additional SPoF2. Scalability issues3. Issues with certificates4. Multiple envs with multiple versions of manifests5. Temptation to use PuppetDB and other, which

add points of failure

4.1 Next steps: Get rid of Puppet Master

● Controller role○ nova-api○ nova-scheduler○ MySQL <- what if customer wants it on the other

server?○ glance-api○ glance-registry○ keystone

4.2 Next steps: Deployment by stages

DB

Controller

node 1 node 2

[{ ‘role’: ‘nova-api’, ‘depends’: [‘MySQL’, ‘keystone’],

{ ‘role’: ‘keystone’, ‘depends’: [‘MySQL’] ]

4.2.1 Dependency graph

MySQL

Keystone

nova-api

4.2.2 Allow other tools for stages

Such as, but not limited to:

● Opscode Chef● SaltStack● Just Ruby or Python code

● http://docs.mirantis.com/fuel-dev/● https://github.com/Mirantis/fuel/pull/572/files● http://nuknad.com/2011/02/11/self-

classifying-puppet-nodes/● https://github.com/stackforge/fuel-{main, web,

astute, ostf}● http://puppetlabs.

com/presentations/continuously-integrated-puppet-dynamic-environment

Links

Mike Scherbakov,

mscherbakov@mirantis.com

Questions?