living system or build factory - chris maxwell

Living System or

Build Factory?!

Chris Maxwell !

@WrathOfChris

Cloud guy @ Ubiquiti Networks

Who is this talk for?

• We build systems

• We turn design into implementation

• We operate systems forever**not forever

We keep systems from failing

Holding up the system?

We do it wrong

Not helping

What is a Living System?

The business has a need

Instructions are unclear

What about THIS space?

Instructions are not followed

...except a VCR, CD’s, and a remote?

Production loads change unexpectedly

Additional capacity added

Wrong tools at hand

12ga crimp on 24ga wire doesn’t work

Patches are put into production

Patches become production

Production is unexpected

Unexpected Scotsman

Parts not available

No ventilation ducting available

Inconsistent Origins

One of these things

is not like the other

http://www.flickr.com/photos/msittig/4801606785/

• Has a long life

• Mutates over time

• Each system slightly different

• The system is the state

Living System

http://www.flickr.com/photos/anijdam/2362660336/

Problems!

Can you rebuild it … fast?

… accurately?http://www.flickr.com/photos/adactio/2856955322/

Problems!

Do you keep: • HA failover? • Standby host?

Problems!

Have you tested? !

When both fail?

What is a Build Factory?

For business…:

Run the linehttp://www.flickr.com/photos/pasukaru76/6893926948/

For devs…:

make clean && make

For ops…:

Kill it with firehttp://www.flickr.com/photos/comedynose/3549464622/

Looking back on Build Factory

Copying systems

Cloning systems

Early build factories took time

...hours on 10baseT

...tumbleweeds...

It was a manual process

How to restore a VM image...

...in 7 easy pages of screenshots

Sorry, couldn’t release these pics :( It was just lots of screenshots

+ do this, then that instructions

It was a manual process

...and still manual work afterhttp://spalinux.com/wp-content/uploads/2008/10/

windows_join_domain.png

Scripting is a start

I will replace you with a short script

Makefiles track dependencies

or a Makefile

it builds your kernel, and probably your OS

Today’s tools

Not a complete list

Describe

Write down all of the steps

http://www.flickr.com/photos/spree2010/4960430323/

Automate

“too complex for any one man’s memory”http://www.flickr.com/photos/sepblog/3679949333/

Replicate

Build 3 the same, tie them together

Repeat

Training makes elements replaceable

Build Factory

• Has a short life

• Replaced for each change

• Each system the same

• The procedure is the state

http://upload.wikimedia.org/wikipedia/commons/5/50/Krispy_Kreme_Doughnuts.jpg

When something jams !

deep in the system

a scrap of paper caught in the drive gear

Replace It!•fix it later •send it for

repair •get production

working!

When production patches take over

Production maintenance changes the system

Rebuild it!

Sometimes you get to rebuild them

In the cloud?

!

live replace might cost up to 1 hour

of billing

In the datacentre?

!

live replace can be done with

spare capacity

Takeaway

• Describe it

• Automate it

• Produce it

• Repeat it

Questions?!

@WrathOfChris !

chris@wrathofchris.com !

Like the Loch Ness Monster,Answers are over there...

• Forgotten attributions: • ghost-1:http://www.all-internet-security.com/images/pc-copy.jpg

• ghost-2:http://www2.expta.com/uploaded_images/Ghost-741732.png

• ghost-3:http://internship.wesmilliron.com/wp-content/uploads/2012/08/vmware-norton-ghost.jpg

• laplink: http://thumbs1.ebaystatic.com/d/l225/m/mkhsfLBMjkjjMw9skfQUNag.jpg

• lap link:http://www.old-games.com/screenshot/5736-1-lap-link-3-for-dos.jpg