Dan Bode| Puppet Labs dan@puppetlabs.com

Hacking The Data out of Puppet

# puppetconf # puppetize @ puppetlabs

Who is this talk for?

•  People who already understand Puppet

•  Developers or people who are dev-curious

# puppetconf # puppetize @ puppetlabs

Shameless plug

# puppetconf # puppetize @ puppetlabs

What is it about?

•  Deconstructing Puppet to data

Dissecting a Puppet Run

Puppet as Data

# puppetconf # puppetize @ puppetlabs

Facter, who am I?

Agent

Hi! your facts are: kernel=linux ipaddress=10.0.0.3 macaddress=…

# puppetconf # puppetize @ puppetlabs

Hi Mr. Master, I need a catalog. Here

are my facts

http://www.dgcomputers.org/testimonials.php

Agent

facts

# puppetconf # puppetize @ puppetlabs

facts

Agent Thanks for you facts. I’ll just store them in

PuppetDB

PuppetDB

# puppetconf # puppetize @ puppetlabs

PuppetDB

facts

ENC

Agent

Mr. ENC, is this host defined as an external

node? Yep, he should be an

apache server. Here is the definition

nodes

# puppetconf # puppetize @ puppetlabs

PuppetDB

facts

Agent Just compiled your

catalog. One sec while I store it in PuppetDB.

catalog

# puppetconf # puppetize @ puppetlabs

PuppetDB

facts

Agent

catalog

Here is your catalog. Send me a report and let me know how it

went!

catalog

# puppetconf # puppetize @ puppetlabs

PuppetDB

facts

Agent

catalog

I hate to be a bother, but can you compute

the md5sums of a few files?

catalog

# puppetconf # puppetize @ puppetlabs

PuppetDB

facts

Agent Just finished applying. Here are the results.

report

catalog

Interacting with Puppet’s Data

Termini and the indirector

# puppetconf # puppetize @ puppetlabs

facts find from terminus facter

Agent

# puppetconf # puppetize @ puppetlabs

catalog find from terminus rest

http://www.dgcomputers.org/testimonials.php

Agent

facts

# puppetconf # puppetize @ puppetlabs

facts

Agent facts save to terminus

puppetdb

PuppetDB

# puppetconf # puppetize @ puppetlabs

PuppetDB

facts

ENC

Agent

node find from terminus exec (or

ldap)

nodes

# puppetconf # puppetize @ puppetlabs

PuppetDB

facts

Agent catalog find from

terminus compiler

catalog

# puppetconf # puppetize @ puppetlabs

PuppetDB

facts

Agent catalog save to

terminus puppetdb

catalog

# puppetconf # puppetize @ puppetlabs

Facter ENC

Disecting a Puppet Run

Com

piler

Config Catalogs

Nodes/Manifest

Reports

Facts

# puppetconf # puppetize @ puppetlabs

CLI commands

puppet facts find

puppet node find

puppet catalog find

# puppetconf # puppetize @ puppetlabs

CLI Puppet Facts

# mkdir –p /tmp/yaml/facts

# puppet facts find node_name --render-as yaml \

> /tmp/yaml/facts/node_name.yaml

# puppetconf # puppetize @ puppetlabs

Creating a node (optional):

# puppet node find node_name \

--node_terminus=exec \

--external_nodes=/etc/puppet/nodes.sh \

--facts_terminus=yaml \

--clientyamldir=/tmp/yaml/ --render-as=yaml \

> /tmp/yaml/nodes/node_name.yaml

# puppetconf # puppetize @ puppetlabs

Creating a catalog:

# puppet catalog find node_name \

--facts_terminus=yaml \

# puppet catalog find node_name \

--node_terminus=yaml \

--manifest=/etc/puppet/manifest/site.pp \

--modulepath=/etc/puppet/modules/

--clientyamldir=/tmp/yaml/ > /tmp/catalog.yaml

# puppetconf # puppetize @ puppetlabs

Fun with IRB

Puppet::Node::Facts.indirection.find facts

Puppet::Node.new nodes

Puppet::Catalog.indirection.find catalog

# puppetconf # puppetize @ puppetlabs

IRB Facts

irb:> require ‘puppet/face’

> facts=Puppet::Face[:facts, :current].find('node’)

# puppetconf # puppetize @ puppetlabs

Access a Fact value (irb):

…

> facts.values['ipaddress']

=> "10.0.2.15"

# puppetconf # puppetize @ puppetlabs

Creating a node (from irb):

…

> node=Puppet::Node.new('node_name',

{:classes => {:foo => {:bar => :baz}}})

>node.merge(facts.values)

# puppetconf # puppetize @ puppetlabs

Creating a catalog:

…

irb> catalog=Puppet::Resource::Catalog.indirection.\

find('node_name’, :node => node)

Interacting with Puppet’s Data

Use Cases

# puppetconf # puppetize @ puppetlabs

Inspecting the catalog:

•  What types are in the catalog?

irb> catalog.resources.collect do |r| r.type end.uniq

•  Gimme a resource:

irb>catalog.resource(‘Package[httpd]’)

# puppetconf # puppetize @ puppetlabs

Rspec Puppet:

let :facts do

{:operatingsystem => ‘Redhat’}

end

let :params do

{:bind_address => ‘0.0.0.0’

end

it { should contain_file(‘/tmp/foo.conf’) }

# puppetconf # puppetize @ puppetlabs

Thundering Herd

Pre-compile catalogs for faster auto-scaling

# puppetconf # puppetize @ puppetlabs

Applying pre-compiled catalogs:

puppet apply --catalog /tmp/catalog.json –server

puppet-fileserver

# puppetconf # puppetize @ puppetlabs

DMZ

tcp over USB

Interacting with Puppet’s Data

Use Cases

# puppetconf # puppetize @ puppetlabs

Hacking reports

Everything in Puppet is a state transition

User[‘dan’] : absent -> present

User[‘dan’][‘shell’] -> ‘/sbin/nologin’ -> /bin/bash

# puppetconf # puppetize @ puppetlabs

Setting up the agent:

[agent]

report=true

# puppetconf # puppetize @ puppetlabs

Archive reports in your yamldir

[master]

reports = store

# puppetconf # puppetize @ puppetlabs

Puppet reports

$ cd `puppet config print reportdir`

$ ls

node1 node2 node3

$ ls node1

# puppetconf # puppetize @ puppetlabs

Every report from every run ever

$ ls node1

201206060256.yaml 201206060303.yaml 201206060519.yaml 201206122349.yaml 201206122354.yaml 201206130002.yaml

# puppetconf # puppetize @ puppetlabs

Lets crack one open!

Irb > require ‘yaml’

>reports=YAML.load_file('201206130002.yaml')

# puppetconf # puppetize @ puppetlabs

Have a look

>(reports.methods - Object.methods).sort

Notice the following methods:

# puppetconf # puppetize @ puppetlabs

High level data

> reports.exit_status

⇒ 0

> reports.status

=> "unchanged"

> reports.host

⇒ ”node1”

# puppetconf # puppetize @ puppetlabs

metrics

> reports.metrics.keys

⇒ ["resources", "events", "changes", "time"]

> reports.metrics['resources']

⇒ [‘failed’, 0],[ ‘changed’, ‘7’]

# puppetconf # puppetize @ puppetlabs

And the awesome sauce

> reports.resource_statuses.keys

=> ["Package[xinetd]", "File[/srv/node/1]", "Package[swift]", "Exec[compile fragments]", "Package[swift-container]", "File[/var/opt/lib/pe-puppet/concat/_etc_swift_object-server.conf]", "File[/etc/rsync.d/frag-account]”]

# puppetconf # puppetize @ puppetlabs

And the awesome sauce

> status = reports.resource_statuses

> status.keys

=> ["Package[xinetd]", "File[/srv/node/1]", "Package[swift]", "Exec[compile fragments]", "Package[swift-container]", "File[/var/opt/lib/pe-puppet/concat/_etc_swift_object-server.conf]", "File[/etc/rsync.d/frag-account]”]

# puppetconf # puppetize @ puppetlabs

And the awesome sauce >events = status["File[/etc/swift/swift.conf]"].events

> events.first.status

⇒ "success”

> events.first.desired_value

⇒ :present

> events.first.previous_value

=> :absent

Thank You Dan Bode| Puppet Labs dan@puppetlabs.com