puppet camp paris 2016 data in modules

Copyright example42 GmbH - 2016

Puppet 4 - Data in ModulesPuppetCamp Paris 2016

Martin Alfke - [email protected]

Image: Tatlin - tatlin.net


mailto:[email protected]


Martin Alfke!Berlin/Germany !CEO example42 GmbH Freelance Puppet Expert Network !Puppet since 2007 !Puppet Trainer, Consultant !Co-Author of “Puppet 4 Essentials”


Puppet 4 Data in Modules

• Separation of Code and Data

• Data in Modules

• Lookup Priority

• Data in Component Modules

• Data in Environments


Separation of Code and Data




• data in code class my_ntp {!!if $::environment == ‘dev’ {! $ntp_server = [‘pool.ntp.org’]! } else {! if $::facts[‘fqdn’] == ‘ntp1.example42.com’ {!# switch back to ntp1 when issue is solved! $ntp_server = [‘ntp2.example42.com’]! } else {! $ntp_server = [‘127.0.0.1’]! }!}!



• explicit lookup

• hiera(‘key’[, ‘default’][, ‘override hierarchy’])



• implicit lookup class my_ntp (!!Array $ntp_server,!) {!!# …!}!!contain my_ntp!!# hiera data!my_ntp::ntp_server:! - ‘pool.ntp.org’!



• hiera.yaml # version 1!:backends:! - yaml!:yaml:! :datadir: “/etc/puppetlabs/code/environments/%{environment}/hieradata”!:hierarchy:! - “nodes/%{::trusted.certname}”! - “os/%{::facts[‘os’][‘osfamily’]}”! - common!



• hieradata os/Debian.yaml apache::pkgname:! - ‘apache2’! - ‘apache2-ssl’!!os/RedHat.yaml apache::pkgname:! - ‘httpd’!!common.yaml apache::purge_configs: true!



• hieradata os/FreeBSD.yaml apache::pkgname:! - ‘apache’!



• puppet code # apache/manifests/params.pp class apache::params {! case $::operatingsystem {! ‘Debian’: { # … }! ‘RedHat’: { # … }! default: {! fail(‘OS not supported’)! }! }!}!


Data in Modules



Lookup Priority



Lookup Priority• Hiera -> Global Overrides

!

!

!



!

• Environment Data -> Core Puppet Lookup

!



!

• Environment Data -> Core Puppet Lookup

!

• Module Data -> Defaults


Lookup Functions• Explicit lookup: lookup(‘key’)!

!

• CLI lookup: puppet lookup ‘key’

!

• Automatic lookup: ‘<namespace>::<key>’


Lookup Functions• lookup(‘key’, <Type>, <merge_behavior>, <default>)!

• e.g. lookup(‘ntp_servers’, Array)

• Merge behavior:

• first!

• unique (array merge)!

• hash!

• deep!


Data Provider Configuration

• Global Environment Provider: puppet.conf

• environment_data_provider = <data provider>!

!

!

!





• Environment Provider: environment.conf


!





• Environment Provider: environment.conf


• Module Provider: metadata.json

• “data_provider”: “<data provider>”


Data Provider

• none -> standard hiera lookup

!

• hiera -> hiera lookup (hiera v4)

!

• function -> data function lookup


Data Provider Hiera

• replace hiera, hiera_array, hiera_hash with ‘lookup’

• needs hiera.yaml v4 configuration file

• set data_provider to ‘hiera’ in puppet.conf, environment.conf or metadata.json

• modify global hiera.yaml to use datadir outside environment


Data Provider Hiera

# /etc/puppetlabs/code/environments/production/hiera.yaml # /etc/puppetlabs/code/environments/production/modules/<module>/hiera.yaml # - - -!version: 4!datadir: hieradata!hierarchy:! - name: “Nodes”! backend: yaml! path: “nodes/%{trusted.certname}”! - name: “OS”! backend: json! path: “os/%{facts.os.family}”! - name: “common”! backend: yaml


Data Provider Function

• write data function

• Puppet 4 Function

• <module>/functions/<module>/data.pp

• <env>/functions/<env>/data.pp

• set data_provider to function in puppet.conf, environment.conf or metadata.json


Data Provider Function - Puppet

# ntp/functions/ntp/data.pp function ntp::data() {! $params = {! ‘ntp::ntpservers’ => [‘pool.ntp.org’],! }! $os_params = case $facts[‘os’][‘family’] {! ‘Debian’: {! { ‘ntp::ntpackage’ => ‘ntpd’, }! },! default: {! {}! }! }! $params + $os_params!}


Data Provider Function

• write data function

• Ruby Function (Puppet 4 function API)

• <module>/lib/puppet/functions/<module>/data.rb

• <env>/lib/puppet/functions/<env>/data.rb

• set data_provider to function in puppet.conf, environment.conf or metadata.json


Data Provider Function - Ruby

# ntp/lib/puppet/functions/ntp/data.rb Puppet::Functions.create_function(:’ntp::data’) do! def base_data()! { ‘ntp::ntpservers’ => [‘pool.ntp.org’], }! end! def os_data()! case Facter.value(:os)[‘family’]! when ‘Debian’! { ‘ntp::pkgname’ => ‘ntpd’, }! else! {}! end! def data()! self.base_data.merge!(self.os_data)! end!end


Data in Component Modules



Data in Component Modules

• add data provider to metadata.json

• provide OS defaults

• remove params.pp / remove inheritance

• allow users to overwrite any data


Data Provider Function - Ruby

# my_ntp/manifests/init.pp class my_ntp (! $server = $my_ntp::params::server,! $pkgname = $my_ntp::params::pkgname,! $secure = $my_ntp::params::secure,!) inherits my_ntp::params {! # ...!}!


Data in Environments



Data in Environments

• old hiera replacement

• add hiera.yaml to environment base path

• overwrite data from modules, roles & profiles


Summary



Summary# /etc/puppetlabs/code/ ! ! hiera.yaml!! ! hieradata/!! ! environments/production/ ! ! ! ! ! ! environment.conf!! ! ! ! ! ! hiera.yaml!! ! ! ! ! ! hieradata/! modules/my_module/ ! ! ! ! ! ! ! ! ! ! metadata.json!! ! ! ! ! ! ! ! ! ! hiera.yaml!! ! ! ! ! ! ! ! ! ! hieradata/


Summary# /etc/puppetlabs/code/ ! ! hiera.yaml!! ! hieradata/!! ! environments/production/ ! ! ! ! ! environment.conf!! ! ! ! ! lib/functions/data.pp!! ! ! ! ! lib/puppet/functions/ \!! ! ! ! ! ! ! ! production/data.rb! modules/my_module/ ! ! ! ! ! ! !! ! metadata.json!! ! ! ! ! ! ! ! lib/functions/data.pp!! ! ! ! ! ! ! ! lib/puppet/functions/ \!! ! ! ! ! ! ! ! ! ! ! my_module/data.rb


Summary - Pro

• Per hierarchy Hiera Data backend possible

• Data Function lookups without need for hiera backend (e.g. Cloud Management API data)

• No more inheritance required


Summary - Con

• No single Source of Authority?

• Debugging can be complex when iterating over many data providers and hierarchies


Module Developers

• switch to data in modules

• give users the possibility to provide own data

• allow users to overwrite any data

• allow users to know their data for missing OS support


Module Users• switch to hieradata in modules

then

• switch to data in environments

• keep data simple and readable

• don’t overcomplicate !


Module Users• hieradata

common.yaml my_ntp: ‘pool.ntp.org’!apache::default_mods: false!apache::purge_configs: true!mysql::remove_default_accounts: true!mysql::root_password: ‘puppet’!oradb::database::version: ’12.1’!oradb::shout: ‘MISSING DATA’


References

• http://docs.puppetlabs.com/puppet/4.3/reference/lookup_quick.html

• http://docs.puppetlabs.com/puppet/4.3/reference/lookup_quick_module.html

• http://puppet-on-the-edge.blogspot.de/2015/01/puppet-40-data-in-modules-and.html


Puppet 4 - Data in modulesPuppetCamp Paris 2016

Martin Alfke - [email protected]



mailto:[email protected]

puppet camp paris 2016 data in modules

Internet