the installshield of the 21st century – theo schlossnagle

ChefThe InstallShield®

of the 21st Century

InstallShield is a registered trademark of Flexera Software LLCNeither Circonus nor Chef or their related companies have or make any claim thereto.

Monday, May 6, 13

Hi, I’m @postwaitMy background is in computing systems engineering

hardware debugging

cut kernel code

network debugging

storage debugging

cut user-space code

operating system release managment

Monday, May 6, 13

Circonus is...

An API-accessible, self-service, state-of-the-art, scalable monitoring and telemetry analysis platform.

With Chef cookbooks available to make monitoring your architecture elegant and simple.

Run as a SaaS... sometimes.

Monday, May 6, 13

This talk is...

not really about the Circonus service.

Monday, May 6, 13

This talk is...

about the challenges involved withturn-key software installation.

Monday, May 6, 13

Installing software

Installing (most closed and some open) softwareis easy.

RPM, deb, IPS, etc.(they all suck, but they all work passably well)

<pkgcommand> install <pkgname>e.g. on OmniOS:

pkg install chef

Monday, May 6, 13

Configuring Software

Varies widely from product to product.

This sucks.

This will never change.

Period.

Monday, May 6, 13

Operating Software

Is actually not that hard.

Most software that survives consumers, runs.

Except when it doesn’t.

This is why we monitor and measure things.

Monday, May 6, 13

What if...

Your software was a complex distributed system?

Your software was the monitoring system itself?

Your software wasn’t your SaaS,but instead your customer’s SaaS on their IaaS.

Monday, May 6, 13

We didn’t start here...

Our software started as SaaS only.

We ran the one, true copy.

Turns out there is still a strong business model around selling enterprise software thatcompanies run on-premise.(in their own cloud.... whatever).

Monday, May 6, 13

Right tool for the job...

In SaaS, we take the “right tool for the job” seriously.

In shipped software, we historically have not.

integration costs are high

support costs are high

licensing challenges

Monday, May 6, 13

A brief look at Circonus

PostgreSQL/pg_amqp, RabbitMQ, redis, memcached, Apache/mod_perl, Node.js, ElasticSearch, OpenSSL (ca)

CEP system (Ernie), case management system (Bert), real-time OLAP system (Razalbath), websockets/etc. (Enzo), metric storage (Srollup & Snowth), API services, web portal, broker (noitd), metric transit (stratcond), long-tail storage services

Monday, May 6, 13

More than 20asynchronous components

Everything can go wrong...it’s just like a Internet infrastructure

Despair.

Monday, May 6, 13

How Chef helped

First: Chef did not save the day.

It did help us quite a bit.

It provided a framework forconverging on an expected state.

Monday, May 6, 13

Chef...

Solo.

Think about it.

It just make sense.

Monday, May 6, 13

1 databag to rule them all

One databag ‘site.json’ that describesthe global topology of all that is Circonus.

It exposes all tunables our clients can control.

A special role called ‘self-configure’/opt/circonus/bin/run-chef self-configure

builds a node file with appropriate roles

Monday, May 6, 13

Chef templating...

Services need to know about other services

site.json has what and where

Chef uses that to template outall the configs forall the services runningon the current node

Monday, May 6, 13

Chef’s upsides

Chef knows how to start services andhow to restart them if they are disabled/in maintenance(this is crucially important in distributed systems)

It means that all dependent services canself-recover simply through diligence.

Monday, May 6, 13

Chef’s upsides

As everything is “automated,” the system is far less tolerant of procedures that sometimes don’t work.

It has the effect ofautomating the QA aroundinstallation and maintenance tasks

Monday, May 6, 13

Chef’s downsides

It has been horribly impracticable to

perform good upgrades (due to packaging)

act as omnipotent state control (due to cost)

Monday, May 6, 13

Chef’s downsides

We support OmniOS and Linux

I want to do the same thing on both:

Apps should live in their own filesystem, logs in another, data in another, use a filesystem for everything. And, of course, use ZFS.

Instead of role: “ernie”

we have roles: “ernie,” “ernie-omnios”, “ernie-rhel”

Monday, May 6, 13

Summary

Chef sucks. Long live Chef.

The process of automating through Chef has:

improved the quality of our deployment process

made it possible to ship a software platform to clients

that runs all the same bits as our production SaaS

and stays up-to-date with our latest bits

Monday, May 6, 13

Thanks

Eric Sproul ~ Circonus Release Engineer(who implemented all of these things)

Monday, May 6, 13