Brian BrazilFounder

Ansible at FOSDEM

Who am I?Engineer passionate about running software reliably in production.

● CS degree from TCD● Contributor to many open source projects, including Prometheus, Python,

Aurora and Zookeeper.● Google SRE for 7 years, working on reliable systems and monitoring● Founder of Robust Perception, providing commercial Prometheus support

and scaling consulting

The Ansible connectionNeeded a configuration management system for Boxever.

Evaluated Puppet, Chef, Salt and Ansible - Ansible was clear winner.

Almost everything converted to ansible.

I use ansible at home.

Several contributions: Worked on “always” tag, monit module and wrote dpkg_selections module.

FOSDEMOpen Source conference in Brussels.

Started in 2000, free to attend.

Thousands of attendees, live streaming, 10k+ devices on wifi.

Using a mix of existing Brussels university infrastructure and it’s own

In 2015 network was rushed, wanted to avoid that this year

2016 Plan

● Get infrastructure setup well in advance● Automate it and check it in● Produce a “conference in a box”

RichiH was hanging around #prometheus, and recruited me and some others to help setup both monitoring and infrastructure.

Starting offTwo weeks in advance we started hacking.

Had a basic working setup two days beforehand, including DNS, Prometheus, SNMP monitoring, SSL, Docker etc.

Collaboration via Github.

At the eventContinued improving the system.

Adding more machines and services was pretty easy, rapid iteration.

Ran into a few issues.

Adding a 2nd completely different type of machine (video boxes, ARM) trickier.

Things to keep in mind: #1 Module bugsIf you run into a bug in a module, check the ansible issue tracker.

There’s probably already a patch there that you can grab.

You can put the fixed version in the ‘library’ directory while waiting for it to get into a proper ansible release.

We had a problem with ‘unarchive’ and URLs.

Things to keep in mind: #2 Galaxy rolesThe quality of Ansible Galaxy roles can vary substantially.

Can also complicate what would otherwise be 3-4 modules when done by hand.

Choose your Galaxy roles carefully.

I’d advise using them for reference (oh, that’s what that setting is called) and maintaining your own consistent set of roles; rather than a mish-mash of Galaxy roles you have to reverse engineer to understand.

Things to keep in mind: #3 Bootstrapping and OrderMake sure when you add new modules and roles that you have dependencies before other modules that need them.

E.g. adding APT repos that require HTTPS before the relevant package has been installed.

Make sure that your roles always work when applied to a fresh machine.

Have a “base” role that does the setup that’s the same on every machine.

Things to keep in mind: #4 Spurious handlersIdempotency is a key value of ansible, if things are already correct don’t change them.

Handlers that fire on every run are annoying, and slow you down a lot as you want for things to restart/reload.

Also make you anxious - did you just push out some other change?

Avoid things like timestamps in files when using the ‘template’ module, so handlers only fire if there’s real changes.

Things to keep in mind: #5 Missing notifyMake sure you have a notify on everything that changes how your services run.

Init files, systemd unit files, configuration files, upgraded deb/rpms, new binary versions.

Having to restart things by hand is annoying.

Use flush_handlers if a subsequent module needs the change.

SummaryAnsible is great for managing machines and collaborating, and also very easy to learn.

A well architected role, handler and variable layout is essential.

Consistency keeps you sane. Configuration is code.

ResourcesFOSDEM: https://fosdem.org

Github: https://github.com/FOSDEM/infrastructure/tree/master/ansible

Talk: https://fosdem.org/2016/schedule/event/fosdem_infrastructure_review/

Robust Perception Website: www.robustperception.io

Ansible example: https://github.com/RobustPerception/demo_prometheus_ansible