puppet camp chicago 2014: running multiple puppet masters (beginner)

Jeffrey MillerSr. Systems AdministratorThe University of Iowa: ITS – EI – SST – ITEmail: Jeff-L-Miller@uiowa.eduFreenode: millerjl1701

� Started VAX mainframes, Sun SPARC systems in college

� Chemistry/Physics teacher… WGS Pro Linux� In 2001, sys admin for the UI Department of

Chemistry doing various things� Now, a infrastructure admin for the UI campus IT

group doing cross-platform stuff� Puppet infrastructure� Red Hat Satellite / Spacewalk� SC: OM, VMM

� 22+ Years in the MN ARNG going various places

� Yes, without a doubt absolutely… maybe

� No infrastructure required other than content distribution… which you probably already have..

� puppet apply….� All nodes have access to the modules and

manifests for your environment� Multiple zones? Multiple data centers? � Configuration churn?

� Test Driven Development administration…

� All machines have access to the entire environment… perhaps your security office has some concerns?

� Reporting is limited� No exported resources� Exported resources allow nodes to share

information with each other� Infrastructure as code

� Can’t I just get by with a single puppet master?

� Yes, without a doubt absolutely… maybe…

http://docs.puppetlabs.com/learning/agent_master_basic.html

� https://docs.puppetlabs.com/puppet/� Seriously, the docs are great!� Pro Puppet. 2013. Krum, van Hevelingen,

Kero, Turnbull, and McCune

� This is the holy grail of running puppet open source….

� maybe…

� Should the servers be physical or virtual?� If you have an virtualization infrastructure

already, use it… � If you don’t have one, build it and then use

it… perhaps puppet can help you with that…

� Leverage redundancy and resilience of the VMware infrastructure

� Ability to scale as needed quickly as more systems are deployed with puppet

� Flexibility in providing puppet infrastructure to non-central units

� Flexibility to test and roll through upgrades as they come out from PuppetLabs

� Java application (ok… it’s clojure inside a JVM… just ask Deepak) that data generated by Puppet with a postgresql database backend

� Stores the most recent set of facts, most recent catalog and by default 14 days of reports

� Provides a very robust API for access to information

� Great for exported resources! (don’t even think about using storeconfigs)

� Really? Hmmmm….� https://docs.puppetlabs.com/guides/scaling_

multiple_masters.html� To scale beyond a certain size � Geographic distribution / disaster recovery� Administrative boundaries (politics)

� There isn’t a golden “though shall must always have more than one at this level”

� Number of resources declared in manifests and infrastructure?

� Exported resources?� Number of nodes?� Time to compile catalog?� Time between runs?� How are other administrative groups using

puppet?

After adding hosts via pdsh After random distribution reestablishes

� Methods for getting the manifests modules out to the puppet masters� Software distribution (RPM, Jenkins testing, etc.)� Puppetfile management with R10K, puppet- librarian,

puppet-librarian-simple� Puppet librarian� Run your own forge! (Pulp)� git pull via cron jobs…� NFS share…

� Puppet 3.6 with the new path is a potential issue for migration in this module…

� LB: 1 core, 2 GB ram� PM-CA: 1 core, 2 GB ram� PM: 4 cores, 4 GB ram� PuppetDB: installed on

each puppet master� PostgreSQL: 2 cores, 8

GB Ram

(VMware Hosts are ProLiant DL385 G7s w/ AMD 6176 2.3 GHz procs)

http://www.reddit.com/r/linux/comments/18weoo/i_manage_a_1600_node_puppet_configuration_ama/

� theForeman and Dashboard use the same postgresql DB backend (2 cores, 8 GB Ram)

� theForeman web frontend: 2 cores, 4 GB Ram� Dashboard frontend: 2 cores, 8 GB Ram (have

puppet master and puppetdb installed on same server for reporting access)

� YMMV on any of these specs

� LB (mod_proxy) splits traffic to appropriate PM

� Puppet Master (PM) can serve out a single or multiple environments via apache/mod_passenger

� The CA CRL is distributed to the LB and PMs

� All reports are forwarded to foreman and dashboard

� Shared file bucket across all puppet masters via NFS

� Leverage redundancy and resilience of the VMware infrastructure

� Can quickly scale as needed for systems or environments

� Security and partitioning of environments

� Safely test and roll through server upgrades

� Fast resource additions to reduce chokepoints

� A single load balancer in front (hardware or software)

� A single server running PuppetDB

� A single server running postgresql

� Reporting?

� Nope…� Statically assign masters to the clients in the

puppet.conf file� Use Round-Robin DNS� DNS SRV Records

� Running puppet as a service for multiple groups with different administrative boundaries

� Everything is just peachy for getting manifests out and reports separated with RBAC! except…

� PuppetDB: All for one and one for all… full access to any facts, catalogs, and reports in the database… no RBAC

� Questions?� Comments?� Divine revelations?