2014-10-28 puppetnyc meetup: managing externally generated files with puppet
DESCRIPTION
This presentation was given at PuppetNYC - New York Puppet Users Group on October 28, 2014. Collective faced a challenge when trying to manage Kerberos keytabs with Puppet: a file{} resource on its own doesn't execute the command needed to build the credentials, and an exec{} resource was neither desirable nor sufficiently secure. This presentation will detail how Collective addressed these limitations and now generates these credentials automatically, securely and efficiently, and how the solution was scaled to multiple Puppet masters using an alternate backend for Hiera. This approach can be generalized to other situations when Puppet needs to manage externally generated files, such as signing website certificates and building replication agreements between LDAP servers. Speaker Bio: Kevin Paulisse is the DevOps Team Lead at Collective, where he and the team use Puppet to manage hundreds of physical and virtual servers including a 300 node Hadoop cluster. Kevin has a 17+ year career including over 3 years as a Puppet devotee. Kevin's experience prior to Collective includes Shopbop.com (part of the Amazon.com family of companies), Philips, the State of Wisconsin, TDS Telecom, Alliant Energy, and his own software company he started while in college.TRANSCRIPT
Managing ExternallyGenerated Files with Puppet
Kevin PaulisseDevOps Engineer and Team Lead
Collective, Inc.
Hello, I am Kevin Paulisse
Background:
• Linux guy for 17+ years, Puppet devotee for 3+ years• Current position: DevOps engineer and team lead @ Collective• Prior: Shopbop (Amazon), Philips, State of WI, TDS, Alliant Energy
Contact information:
• Work e-mail: [email protected]• Personal e-mail: [email protected]• Twitter: @kpaulisse• Github: kpaulisse• LinkedIn: https://www.linkedin.com/in/kpaulisse
[email protected]@gmail.com@kpaulissehttps://github.com/kpaulissehttps://linkedin.com/in/kpaulisse
About Collective
What Collective does:
• Serve Ads: 10's of thousands of requests/sec• Bid on Ads: 100's of thousands of requests/sec• Data Science: 1000's of predictive models weekly
Collective's technology stack:
• Linux (physical & virtual)• Puppet and Foreman• 3 new data centers this year• 300 node Hadoop cluster
Collective is hiring:
• DevOps/Linux Engineers• Data Scientists, Devs, etc.
[email protected]://www.collective.com
Agenda
• Introduction
• Kerberos crash course
• Managing Kerberos keytab files with Puppet:
• Traditional approaches: file{} and exec{}
• Other approaches to generating keytabs
• Our implementation
• Scaling the solution to multiple Puppet masters
• Other use cases
Kerberos crash course: Terminology
Kerberos: Networkauthentication protocol
Principal: An account
Keytab: A file containingencrypted keys forprincipals
Host Keytab: On Linux,/etc/krb5.keytab
Kerberos crash course: Keytab files
Each host needs its own Kerberos principal and corresponding keytab.
A Kerberos Administrator creates and manipulates principals.
Task NeedsAdmin
ScramblesPassword
Determine if there is a Kerberos principal for the host
Create the Kerberos principal if none exists
Save Kerberos credentials to a keytab file
(Scrambling password refers to MIT Kerberos -- other systems may differ.)
Installing Kerberos keytab with file{}
file { "/etc/krb5.keytab": ensure => present, owner => "root", group => "root", mode => 0400, source => WhatGoesHere?}
● The keytab file must already exist somewhere
● The above resource does not generate the file
Installing Kerberos keytab with exec{}
exec { "create-etc-krb5.keytab": command => "kadmin -w S3cr3tPass ...", creates => "/etc/krb5.keytab", user => "root",}
● The exec resource runs on the agent
● Each agent gets to see your administrative credential
Generate keytab asynchronously: Design
Keytab generation
● Use the admin credential on just one server
● Generate keytab before Puppet runs (put keytab where Puppet can find it)
● Store the keytabs in such a way that a host can see only its own keytab
Puppet manifest
● Use a file{} resource to define content/source
● Host keytab in catalog (not admin credentials)
● Usual file{} features: set permissions, restart services, etc.
Generate keytab asynchronously: Implementation
Manual processDocs: "Don't forget to run the keytab script"
Management system (e.g. The Foreman)Generate keytab via a "hook" from management system when a host is added to the configurationor a Puppet run is scheduled
Task: Generate a keytab file before the Puppet run
Generate keytab during Puppet run: Design
Keytab generation
● Keytab has to be generated during the Puppet run
● Keytab has to be available on Puppet master so it can be included in the catalog
● Don't use exec{} on agent to generate keytab
Puppet agent
● Use a file{} resource to define content/source
● Host keytab in catalog (not admin credentials)
● Usual file{} features: set permissions, restart services, etc.
Internals of a Puppet run
Read Further: https://docs.puppetlabs.com/puppet/3.7/reference/subsystem_agent_master_comm.html
1. SSL certificate generation and signing
2. Plugin sync
3. Build and fetch catalog (pictured at left)
a. HTTPS POST with facts
b. Catalog = everything but file sources
4. Apply (and fetch files where needed)
5. Send reportPOST /catalog
Manifests
Hiera
Compiler
Templates
Send factsReceive catalog
Apply catalog
Keytab generation during Puppet run: Implementation
exec{}
POST /catalog
Manifests
Hiera
Compiler
Templates
Send factsReceive catalog
Apply catalog
ERB template<%= `kadmin -w S3cr3tPass ...` %>
Task: Generate keytab before/during catalog compilation
file { "/etc/krb5.keytab": ... content => template("krb5/keytab.erb")}
Keytab generation during Puppet run: Implementation
exec{}
POST /catalog
Manifests
Hiera
Compiler
Templates
Send factsReceive catalog
Apply catalog
ERB template<%= `kadmin -w S3cr3tPass ...` %>
Puppet generate( ) function
Custom Puppet function
Part of agent certificate signing
Task: Generate keytab before/during catalog compilation
Our implementation: Custom function
PSEUDO CODE
$h_keytab = hiera('krb5-keytab', '*undefined*')
if ($h_keytab == '*undefined*') { $keytab = generate_keytab( { options... } )} else { $keytab = $h_keytab}
file { "/etc/krb5.keytab": replace => true, content => $keytab,}
POST /catalog
Manifests
Hiera
Compiler
Templates
Send factsReceive catalog
Apply catalog$h_keytab = hiera('krb5-keytab', generate_keytab(...))
Our implementation: Custom function
$h_keytab = hiera('krb5-keytab', '*undefined*')
if ($h_keytab == '*undefined*') { $keytab = generate_keytab(...)} else { $keytab = $h_keytab}
file { "/etc/krb5.keytab": replace => true, content => $keytab,}
● Puppet master has admin keytab
Our Implementation Details:
● We store the admin keytab in hiera for easier packaging
Our implementation: Custom function
● Puppet master knows admin keytab
Our Implementation Details:$h_keytab = hiera('krb5-keytab', '*undefined*')
if ($h_keytab == '*undefined*') { $keytab = generate_keytab(...) store_keytab_in_hiera(...)} else { $keytab = $h_keytab}
file { "/etc/krb5.keytab": replace => true, content => $keytab,}
● We store the admin keytab in hiera for easier packaging
● Call a custom function to store the generated keytab in hiera
Our implementation: Custom function
● Puppet master knows admin keytab
Our Implementation Details:$h_keytab = hiera('krb5-keytab', '*undefined*')
if ($h_keytab == '*undefined*') { $keytab = generate_keytab(...) store_keytab_in_hiera(...)} else { $keytab = base64('decode',$h_keytab)}
file { "/etc/krb5.keytab": replace => true, content => $keytab,}
● We store the admin keytab in hiera for easier packaging
● Call a custom function to store the generated keytab in hiera
● Base64 encode keytabs in hiera; decode them when retrieved
Scalability challengeData Center #1 Data Center #2
Puppet Master1A
Puppet Master1B
Puppet Master2A
Puppet Master2B
Node Node Node Node Node Node
CouchDB
CouchDB
CouchDB
CouchDB
Scalability challengeData Center #1
Puppet Master1A
Puppet Master1B
Node Node Node
CouchDB
CouchDB
CouchDB Pros:
● Open source (Apache)
● Native multi-master replication
● Simple HTTP/JSON interface
● Records are versioned
● Web GUI (futon)
CouchDB Cons:
● Renew replication at restart
● No LDAP auth for users
Other use casesCertificate signing for websites
• Generate key and CSRvia script on Puppet Master
• Sign certificate with the CA's private key from Puppet Master
• Store key and certificate in hiera
• Use file{} to install certificates
• Notify service{} to restart web server to pick up new certificate
LDAP server configuration
• Use package{}, service{} etc.in typical fashion
• Use file{} to install configuration files on LDAP server
• Add replication agreement using LDAP admin credential (can run from Puppet Master)
• Notify service{} to restartwhen replication is set up
ResourcesPuppet module:https://forge.puppetlabs.com/collectivemedia/krb5keytabhttps://github.com/collectivemedia/puppet-krb5keytab
Slide deck:https://slideshare.net/kpaulisse/nyc-puppetpresentation
Collective website:
http://www.collective.com
Collective careers:
Contact info:
Kevin Paulisse
Work: [email protected]
Home: [email protected]
Twitter: @kpaulisse / Github: kpaulisse