puppet camp amsterdam

25
Story of Puppet @eBay Classifieds Group or how we came to the setup we use

Upload: vladimir-lazarenko

Post on 12-Jul-2015

447 views

Category:

Documents


0 download

TRANSCRIPT

Page 1: Puppet camp amsterdam

Story of Puppet @eBay Classifieds Group

or how we came to the setup we use

Page 2: Puppet camp amsterdam

Some bio or “Who is Vladimir”

Sysadmin for eighteen years

Currently in Solutions team of eCG’s marktplaats.nl

All-round nerd

Hobbyist programmer

Page 3: Puppet camp amsterdam

Puppet at eBay Classifieds:

9 countries

13 platforms

1 codebase

1 change management process

Page 4: Puppet camp amsterdam

Puppet at eBay Classifieds:

We started adopting puppet in 2008

Since then we had about 4 different iterations of “this is bloated and

doesn’t work, let’s think more and re-do”

Currently we believe we achieved a workable scenario where we:

Enable teams to share and re-use puppet code

Enforce the same code style and standards

Allow for peer-reviews of the code, also enforce them

“If it’s not in Puppet – it does not exist.”

Page 5: Puppet camp amsterdam

What we are going to talk about:

• Different iterations of puppet implementation at eCG

• Go in-depth on the structure of the current one

• Process around the codebase

• Tools

Page 6: Puppet camp amsterdam

“Bucket of recipes”

Puppet at eBay Classifieds (1st implementation):

A bunch of modules without visible structure (server-role based)

Relations between modules on any levels (includes, instantiations)

One Subversion repository, where everyone works on trunk

What were the lessons learned?

As puppet adoption grew, modules for different platforms started

clashing (same, yet a bit different functionality)

As number of modules and their dependencies grew people were

afraid of making changes - dependency tree was very hard to follow

Resulted in tons of duplicate code

Change management was very risky – you commit something and

it’s live

Page 7: Puppet camp amsterdam

“Buckets of recipes”

Puppet at eBay Classifieds (2nd implementation):

What did we improve?

Release management with Subversion branching

Better module grouping

What were the lessons learned?

Waiting on releases is killing. If you need a change - you need it

now, not in 2 or 3 days

Provokes people to lock machines to their own puppet environment

Surprises at times of the releases. For some people it was daytime,

for others it was middle of the night

Page 8: Puppet camp amsterdam

Puppet at eBay Classifieds (3rd implementation):

What did we improve?

Switched to Git

Completely redesigned modules structure (more on this later)

Introduced extlookup for data separation vs configuration logic

Release more often

What were the lessons learned?

When release process is human-dependent it will be a bottleneck

Extlookup is nice, but in many places unreadable

Page 9: Puppet camp amsterdam

Puppet at eBay Classifieds (4th implementation):

What did we improve?

Introduced Gerrit for code reviews

Implemented automatic puppet-lint, template syntax and PEP8 checkers

Introduced automated quorum votes for peer reviews

Release as soon as your code passed checks and YOU need it live

Further improved module tree structure

Replaced extlookup with hiera

Introduced puppetdb

Page 10: Puppet camp amsterdam

So how does it look now?

Shared

eCG

Platform

Config

Four module levels

One include direction

No includes on the same level

Page 11: Puppet camp amsterdam

Shared modules

Shared

eCG

Platform

Config

Shared modules are very basic, deploy one

piece of software in its standard configuration (no

customization)

Standalone. Includes no other modules

Installs needed packages, basically

Parameterized to allow configuration override

Examples: apache, ntpd, activemq, mysql, postgres

Page 12: Puppet camp amsterdam

eCG modules

Shared

eCG

Platform

Config

eCG modules add our specific eCG-wide

approved customizations. Rule: it must play well

with majority of teams to make it

Mostly include shared modules

Never include each other

Override default shared module configuration

with eCG specific one via parameters

May be parameterized to allow for configuration

tweaks or overrides

Examples: puppetmaster, nagios checks, collectd

plugins

Page 13: Puppet camp amsterdam

Platform modules

Shared

eCG

Platform

Config

Platform level introduces company-specific

modules and configuration customizations

May include eCG modules

May include shared modules

Never include each other

Page 14: Puppet camp amsterdam

Config modules

Shared

eCG

Platform

Config

Config level introduces server-role specific

modules

Are the starting point for our servers

configuration

May include platform, eCG or shared modules

Never include each other

Node definition must include config module

Page 15: Puppet camp amsterdam

Why did we do it like this?

Shared

eCG

Platform

Config

Easy starting point. Look up a node, see its

class, open the class – you see on one page

how your server is configured on a global level

If you need to know more, dive deeper

We enable code sharing, we do not enforce it

If you see a module in shared you can use – use

it. If you can’t use it, extend it in a backwards-

compatible way or write other extension on an

appropriate level

Page 16: Puppet camp amsterdam

Hiera

Great way to separate code from data

Company-based structure

Lookup path based on facts to allow company data separation

while keeping the same source tree

:datadir: /var/lib/hiera/%{environment}/%{companyprefix}

Inside company dirs a variety of fact-based lookups to allow for

fine-grained data structures

Page 17: Puppet camp amsterdam

PuppetDB

Much faster catalog compilations for huge catalogs (nagios

server with 200k resources)

REST API to talk to for inventory and searches

Many brilliant ways to (ab)use it

Stores all facts (can be used to feed CMDBs)

Stores most recent catalog (can be used for inspections)

POC we are running - internal Puppet naginator module is very slow,

collection of exported resources takes about 15 minutes. We use

PuppetDB to generate the same nagios configs in just under 8

seconds

“If there is data, it can be used”. PuppetDB is a central data store.

Love.

Page 18: Puppet camp amsterdam

So all good, but how does this work in practice?

Per-user environments on puppetmasters

Make the change in your own env

Test it from your own env

Push to Gerrit

Ask team members to review

Merge

Code reviewPush to GerritTest the change

with --environmentLocal

environmentMerge

Page 19: Puppet camp amsterdam

Gerrit

Great code review tool (https://code.google.com/p/gerrit/)

Acts as a Git server with fine-grained access control

Each commit is a separate change, that requires votes for

approvals

Votes can be +1 for Code Review, +2 for change approval

No +2 – no merge

We use voting system with auto +2

Depending on platform size or number of platforms affected by the

change more votes are needed to get a change through

Each change can be “pushed through” in case of an emergency

Page 20: Puppet camp amsterdam

More Gerrit goodies

Allows for post-push hooks

Puppet-lint

Ruby templates parsing

PEP8 (we love Python)

Each change is separately checked out by hooks and tested

for syntax or other obvious errors

Concept of “one commit – one change” forces people to create

atomic changes that are easy to revert in case of errors

Keeping it all in the same system, even if you decide to split Git

repositories still makes work done by colleagues VISIBLE

Page 21: Puppet camp amsterdam

Gerrit and puppetmasters

Master is automatically pulled by crons on puppetmasters

Gerrit disallows pushing directly to master

Even if you force-merge a change, it makes your change

VISIBLE and easily REVERTIBLE

Takes about half a minute extra time

Ensures you didn’t slack out on syntax checkers

Page 22: Puppet camp amsterdam

To sum it all up

There is no single structure that fits every company

Puppet code is like any other code

Write

Use

Refactor if needed

Layered modules structure can work well in many cases

Complementary tooling around Puppet is a life- (and time-)

saver

Lint(s), Gerrit, PuppetDB

Page 23: Puppet camp amsterdam

Work to be done in our house

Approval process improvements

Better quorum schemas

Easier ways of code style and syntax checking

See that people don’t leave changes “hanging”

Integrate Gerrit with corporate GitHub

Work more with community

Open source some generic shared modules

Offer changes to PuppetForge modules

POC with Vagrant for puppet code testing

Page 24: Puppet camp amsterdam

Questions?

Page 25: Puppet camp amsterdam

Thank you!

You can reach me by:

• Mailing to [email protected]

• Following me on GitHub

(https://github.com/favoretti)

Special thanks to:

• PuppetLabs and eCG for the opportunity to speak here

• Teams of eCG for bearing with me all these years

• You all for attending and listening

We are hiring!

Visit http://ebayclassifiedsgroup.com/careers

for more info.