Our Puppet History
☁ Early Puppet adopters … since version 0.25.X
☁ Large scale environment with distributed team
☁ We made every possible mistake
We’re on our 3rd major Puppet implementation!
Lessons Learned
☁ Do NOT make changes directly on the Puppet Master
☁ DO your testing - lint, code validation, etc.
☁ Do NOT pollute your downloaded official modules
☁ DO define and document the standard workflow
☁ Do NOT store sensitive data into modules
Standardised process for deploying changes
Easily repeatable for multiple
environments
Puppet “V3” Requirements
High Availability
Bring order to our modules
Ingredients
☁ Packer
☁ Puppetlabs R10k
☁ Custom Ruby Service - “Captain Hook”
☁ Notification Systems - Slack/Flowdock
EC2 ASG ELB
SNS SQS RDS
CloudFormation
☁ Module preferences☁ Puppetlabs supported☁ Vendor supported☁ Puppet approved☁ Custom modules (role/profile)
☁ Create module guidelines else it gets very messy!☁ Each module lives in a separate Git repository
Puppet Modules
$ aws cloudformation create-stack \
--stack-name puppetmaster-staging \
--templatebody file://infrastructure-puppetmaster.template \
--parameters \
ParameterKey=Environment,ParameterValue=staging \
ParameterKey=InstanceType,ParameterValue=m3.large \
ParameterKey=MinScalingGroupSize,ParameterValue=2 \
ParameterKey=MaxScalingGroupSize,ParameterValue=4
CloudFormation Example
Awesome, so we have a neat-looking HA/FT set-up…
…but how do we manage pushing changes to Puppet Masters?!
☁ Puppet masters can “come and go” randomly
☁ Keep the modules up to date per environment
☁ K.I.S.S. - Tame the learning curve for the team
☁ Avoid reinventing the wheel
Challenges
1. Make changes and commit/push to git server2. Git server triggers post-commit hook (POST) to Captain Hook server3. Captain Hook server validates the payload & creates new message:
☁ Full refresh ➩ r10k deploy environment -p☁ Light refresh ➩ r10k deploy environment☁ Module refresh ➩ r10k deploy module <name>
4. Captain hook server pushes new message to SNS5. Captain hook client polls & reads messages ( ➩ r10k)6. … and we get notified in slack/flowdock/dashboard
Typical Puppet Workflow
☁ Fairfax is a large scale complex environment
☁ Many systems engineers are constantly pushing changes
☁ Puppet architecture designed for HA and fault tolerance
☁ Puppet workflow helps us deploy changes to multiple
masters safely and easily
Summary