puppet camp melbourne 2014:
TRANSCRIPT
I moved from managing 1 to 100 servers with Puppet
Eric Pizzani The Australian National University
Quick Intro: Me • Works in Canberra for the Australian National
University, originally from Melbourne. • Responsible for the technical infrastructure of
ANU College of Asia and the Pacific’s web presence.
Quick Intro: The College • The college has roots dating back to 1946 as
the Universities’ Research School of Pacific Studies
• 220 academic staff supported by 170 general staff
• 2,252 students with over 9,600 alumni • Subjects from economics, political studies to
Asian languages • Have more IP addresses than North Korea
Quick Intro: The College More to the point… • Hosts over 60 websites • Average of five thousand visitors a day • Peak of 100,000 on one night • Running Drupal, Wordpress, and hand
crafted PHP
In the beginning… • Small IT firm of 4 full time staff and a handful
of contractors • Desktop support to system administration • Configuration management was needed
In the beginning…
“Puppet sucks. Puppet is written in Ruby. Ruby sucks.”
In the beginning…
“Puppet sucks. Puppet is written in Ruby. Ruby sucks.”
– Eric Pizzani, 2009
In the beginning…
In the beginning…
In the beginning…
In the beginning…
What could possibly go wrong?
In the beginning…
In the beginning… Just a bad idea
And repeat every time there’s an update? For every machine?
In the beginning… Just a bad idea
And repeat every time there’s an update? For every machine?
In the beginning…
Fairly sure this sets up P HP in an insecure way…
In the beginning…
As before...
What if this needed to be larger?
In the beginning…
T URN OF F THE F IREWALL!?
Why are they instal led in the f irst place?
In the beginning…
Good start, but I’m sure there’s more to do...
ANU – ‘many’ years later • Used pre-configured virtual appliances • Low monthly fee gave you access to a wide
range of web applications- Wordpress, OTRS, Drupal
• Simple to setup- Download the appliance, fill in a few questions in a form and done.
• Updates were the same process
ANU – ‘many’ years later • Used pre-configured virtual appliances • Low monthly fee gave you access to a wide
range of web applications- Wordpress, OTRS, Drupal
• Simple to setup- Download the appliance, fill in a few questions in a web form and done.
• Updates were the same process ...in theory
ANU – ‘many’ years later
What could go wrong? • Mistakes and typos • Steps forgotten • Poor asset management • Repeating the entire initial configuration
process every time you needed to update was time consuming
What did go wrong? • Custom configuration lost on upgrade or DR. • Wordpress install inside VA provided
wordpress install. • Files ended up in places the VA backup
service didn’t know about…
What else went wrong? • DNS server addresses changed • I would forget this important step
• Systems were frequently not put into Nagios
Enter puppet
Enter puppet
=)
The Project • Legacy Solaris Stack Decommissioning – Management machine was a 1997 vintage Sun
Enterprise 450 – Web server was a 2003 vintage Sun Fire V440 – Custom complied Apache and PHP – Non-existent documentation – 1200 days uptime – 40 websites and sub sites
The Project
The Project
The Project
also known as cheops
DNS
The Project
also known as cheops
DNS SMTP
How puppet helped?
How puppet helped?
How puppet helped?
How puppet helped?
How puppet helped? • Quick site provisioning – No messing and crafting vhosts by hand
• “Self” documentated • Databases created and firewalls setup
Low hanging fruit…or continuous improvement
• Improvements rolled out to all systems- new and old
• Manual tasks no more! • More monitoring checks!
Low hanging fruit…or continuous improvement
• Improvements rolled out to all systems- new and old
• Manual tasks no more! • More monitoring checks!
Low hanging fruit…or continuous improvement
• Improvements rolled out to all systems- new and old
• Manual tasks no more! • More monitoring checks!
Low hanging fruit…or continuous improvement
• Improvements rolled out to all systems- new and old
• Manual tasks no more! • More monitoring checks! • Auto-documentation
Low hanging fruit…or continuous improvement
Low hanging fruit…or continuous improvement
Night works
• Server outages & updates
Night works
• Server outages & updates • Run yum update -y? Or apt-get update && apt-get upgrade?
Night works
• Server outages & updates • Run yum update -y? Or apt-get update && apt-get upgrade?
mco puppet resource exec yum command='/usr/bin/yum update -y' --with-fact osfamily=RedHat!
What’s next?
• Working on a web infrastructure migration project – Leverage Hiera
What’s next?
• Working on a web infrastructure migration project – Leverage Hiera – More git
What’s next?
• Working on a web infrastructure migration project – Leverage Hiera – More git – More monitoring
What’s next?
• Working on a web infrastructure migration project – Leverage Hiera – More git – More monitoring
...for now…
Questions
???
