test driven infrastructure development
DESCRIPTION
"Test driven Infrastructure development" by Tomas Doran at Puppet Camp Barcelona 2013. Learn about upcoming Puppet Camps at http://puppetlabs.com/community/puppet-camp/TRANSCRIPT
![Page 1: Test Driven Infrastructure Development](https://reader037.vdocument.in/reader037/viewer/2022102815/5562b7e3d8b42a09618b46c8/html5/thumbnails/1.jpg)
Test driven Infrastructure development
Tomas (t0m) Doran<[email protected]>@bobtfishhttps://github.com/bobtfishhttps://github.com/youdevise
Thursday, 14 March 13
![Page 2: Test Driven Infrastructure Development](https://reader037.vdocument.in/reader037/viewer/2022102815/5562b7e3d8b42a09618b46c8/html5/thumbnails/2.jpg)
‘Real men’ develop in production!
Thursday, 14 March 13
Repeat again and again. Development cycle SLOOOW.
![Page 3: Test Driven Infrastructure Development](https://reader037.vdocument.in/reader037/viewer/2022102815/5562b7e3d8b42a09618b46c8/html5/thumbnails/3.jpg)
‘Real men’ develop in production!
• Edit / Commit / Push
Thursday, 14 March 13
Repeat again and again. Development cycle SLOOOW.
![Page 4: Test Driven Infrastructure Development](https://reader037.vdocument.in/reader037/viewer/2022102815/5562b7e3d8b42a09618b46c8/html5/thumbnails/4.jpg)
‘Real men’ develop in production!
• Edit / Commit / Push
• Update puppetmaster
Thursday, 14 March 13
Repeat again and again. Development cycle SLOOOW.
![Page 5: Test Driven Infrastructure Development](https://reader037.vdocument.in/reader037/viewer/2022102815/5562b7e3d8b42a09618b46c8/html5/thumbnails/5.jpg)
‘Real men’ develop in production!
• Edit / Commit / Push
• Update puppetmaster
• puppet agent -t
Thursday, 14 March 13
Repeat again and again. Development cycle SLOOOW.
![Page 6: Test Driven Infrastructure Development](https://reader037.vdocument.in/reader037/viewer/2022102815/5562b7e3d8b42a09618b46c8/html5/thumbnails/6.jpg)
‘Real men’ develop in production!
• Edit / Commit / Push
• Update puppetmaster
• puppet agent -t
• Repeat
Thursday, 14 March 13
Repeat again and again. Development cycle SLOOOW.
![Page 7: Test Driven Infrastructure Development](https://reader037.vdocument.in/reader037/viewer/2022102815/5562b7e3d8b42a09618b46c8/html5/thumbnails/7.jpg)
This is insane!
Thursday, 14 March 13
CHOAS and FAIL result when you break each other. Or, MORE likely (this happens twice a day!)
![Page 8: Test Driven Infrastructure Development](https://reader037.vdocument.in/reader037/viewer/2022102815/5562b7e3d8b42a09618b46c8/html5/thumbnails/8.jpg)
This is insane!
• Try it on an 8 person team.
Thursday, 14 March 13
CHOAS and FAIL result when you break each other. Or, MORE likely (this happens twice a day!)
![Page 9: Test Driven Infrastructure Development](https://reader037.vdocument.in/reader037/viewer/2022102815/5562b7e3d8b42a09618b46c8/html5/thumbnails/9.jpg)
This is insane!
• Try it on an 8 person team.
Thursday, 14 March 13
CHOAS and FAIL result when you break each other. Or, MORE likely (this happens twice a day!)
![Page 10: Test Driven Infrastructure Development](https://reader037.vdocument.in/reader037/viewer/2022102815/5562b7e3d8b42a09618b46c8/html5/thumbnails/10.jpg)
This is insane!
• Try it on an 8 person team.
• ‘LOL - I broke puppet’
Thursday, 14 March 13
CHOAS and FAIL result when you break each other. Or, MORE likely (this happens twice a day!)
![Page 11: Test Driven Infrastructure Development](https://reader037.vdocument.in/reader037/viewer/2022102815/5562b7e3d8b42a09618b46c8/html5/thumbnails/11.jpg)
OI!!!
Thursday, 14 March 13
![Page 12: Test Driven Infrastructure Development](https://reader037.vdocument.in/reader037/viewer/2022102815/5562b7e3d8b42a09618b46c8/html5/thumbnails/12.jpg)
OI!!!
OI t0m!!!!
Thursday, 14 March 13
![Page 13: Test Driven Infrastructure Development](https://reader037.vdocument.in/reader037/viewer/2022102815/5562b7e3d8b42a09618b46c8/html5/thumbnails/13.jpg)
OI!!!
OI t0m!!!!
You broke
puppet!
Thursday, 14 March 13
![Page 14: Test Driven Infrastructure Development](https://reader037.vdocument.in/reader037/viewer/2022102815/5562b7e3d8b42a09618b46c8/html5/thumbnails/14.jpg)
OI!!!
OI t0m!!!!
You broke
puppet!
AARRRGGH!!!
Thursday, 14 March 13
![Page 15: Test Driven Infrastructure Development](https://reader037.vdocument.in/reader037/viewer/2022102815/5562b7e3d8b42a09618b46c8/html5/thumbnails/15.jpg)
Lets fix this!
• First, a glossary:
Thursday, 14 March 13
![Page 16: Test Driven Infrastructure Development](https://reader037.vdocument.in/reader037/viewer/2022102815/5562b7e3d8b42a09618b46c8/html5/thumbnails/16.jpg)
Lets fix this!
• First, a glossary:
• mco - mcollective
Thursday, 14 March 13
![Page 17: Test Driven Infrastructure Development](https://reader037.vdocument.in/reader037/viewer/2022102815/5562b7e3d8b42a09618b46c8/html5/thumbnails/17.jpg)
Lets fix this!
• First, a glossary:
• mco - mcollective
• ENC - External node classifier
Thursday, 14 March 13
![Page 18: Test Driven Infrastructure Development](https://reader037.vdocument.in/reader037/viewer/2022102815/5562b7e3d8b42a09618b46c8/html5/thumbnails/18.jpg)
We can do better
Thursday, 14 March 13
This at least lets you develop things independently. Everyone can do dev in their own branch and merge once they have something that doesn’t break _everything_. You can also rebase -i (squash) all the ARGH PUPPET SYNTAX commits.
![Page 19: Test Driven Infrastructure Development](https://reader037.vdocument.in/reader037/viewer/2022102815/5562b7e3d8b42a09618b46c8/html5/thumbnails/19.jpg)
We can do better
• Branch == environment
Thursday, 14 March 13
This at least lets you develop things independently. Everyone can do dev in their own branch and merge once they have something that doesn’t break _everything_. You can also rebase -i (squash) all the ARGH PUPPET SYNTAX commits.
![Page 20: Test Driven Infrastructure Development](https://reader037.vdocument.in/reader037/viewer/2022102815/5562b7e3d8b42a09618b46c8/html5/thumbnails/20.jpg)
We can do better
• Branch == environment
• Branch / Commit / Push
Thursday, 14 March 13
This at least lets you develop things independently. Everyone can do dev in their own branch and merge once they have something that doesn’t break _everything_. You can also rebase -i (squash) all the ARGH PUPPET SYNTAX commits.
![Page 21: Test Driven Infrastructure Development](https://reader037.vdocument.in/reader037/viewer/2022102815/5562b7e3d8b42a09618b46c8/html5/thumbnails/21.jpg)
We can do better
• Branch == environment
• Branch / Commit / Push
• mco puppetupdate
Thursday, 14 March 13
This at least lets you develop things independently. Everyone can do dev in their own branch and merge once they have something that doesn’t break _everything_. You can also rebase -i (squash) all the ARGH PUPPET SYNTAX commits.
![Page 22: Test Driven Infrastructure Development](https://reader037.vdocument.in/reader037/viewer/2022102815/5562b7e3d8b42a09618b46c8/html5/thumbnails/22.jpg)
We can do better
• Branch == environment
• Branch / Commit / Push
• mco puppetupdate
• puppet agent -t --environment xxx
Thursday, 14 March 13
This at least lets you develop things independently. Everyone can do dev in their own branch and merge once they have something that doesn’t break _everything_. You can also rebase -i (squash) all the ARGH PUPPET SYNTAX commits.
![Page 23: Test Driven Infrastructure Development](https://reader037.vdocument.in/reader037/viewer/2022102815/5562b7e3d8b42a09618b46c8/html5/thumbnails/23.jpg)
Sounds good?
•Then you’ll be wanting:
•https://github.com/youdevise/puppetupdate
Thursday, 14 March 13
It’s a bit basic, but then I ripped it out of work internal code at 8am ;)
![Page 24: Test Driven Infrastructure Development](https://reader037.vdocument.in/reader037/viewer/2022102815/5562b7e3d8b42a09618b46c8/html5/thumbnails/24.jpg)
So we fixed it?
Thursday, 14 March 13
![Page 25: Test Driven Infrastructure Development](https://reader037.vdocument.in/reader037/viewer/2022102815/5562b7e3d8b42a09618b46c8/html5/thumbnails/25.jpg)
So we fixed it?
Thursday, 14 March 13
![Page 26: Test Driven Infrastructure Development](https://reader037.vdocument.in/reader037/viewer/2022102815/5562b7e3d8b42a09618b46c8/html5/thumbnails/26.jpg)
Refactoring
Thursday, 14 March 13
Sorry Chris, but when you say ‘refactoring’ - it’s not refactoring unless you have tests.The problem is that you can’t always remember to run the right branch on all the right nodes. Or rather, how do you even know what all the right nodes are? And if you’re hacking on custom functions, or anything using exported resource - WOE
![Page 27: Test Driven Infrastructure Development](https://reader037.vdocument.in/reader037/viewer/2022102815/5562b7e3d8b42a09618b46c8/html5/thumbnails/27.jpg)
Refactoring
• We change things to be consistent across codebase:
• Why did puppet just delete all the firewall rules on the production database?
Thursday, 14 March 13
Sorry Chris, but when you say ‘refactoring’ - it’s not refactoring unless you have tests.The problem is that you can’t always remember to run the right branch on all the right nodes. Or rather, how do you even know what all the right nodes are? And if you’re hacking on custom functions, or anything using exported resource - WOE
![Page 28: Test Driven Infrastructure Development](https://reader037.vdocument.in/reader037/viewer/2022102815/5562b7e3d8b42a09618b46c8/html5/thumbnails/28.jpg)
Refactoring
• We change things to be consistent across codebase:
• Why did puppet just delete all the firewall rules on the production database?
• We don’t refactor:
• Add bugs all the time due to inconsistency
Thursday, 14 March 13
Sorry Chris, but when you say ‘refactoring’ - it’s not refactoring unless you have tests.The problem is that you can’t always remember to run the right branch on all the right nodes. Or rather, how do you even know what all the right nodes are? And if you’re hacking on custom functions, or anything using exported resource - WOE
![Page 29: Test Driven Infrastructure Development](https://reader037.vdocument.in/reader037/viewer/2022102815/5562b7e3d8b42a09618b46c8/html5/thumbnails/29.jpg)
• Hard coded IPs in 10 places
Unfortunate reality:
Thursday, 14 March 13
So, despite our best efforts, our puppet code was SHIIIIT.Exported resources IS NOT a good fit for non-trivial things (like generating load balancer configs). Ergo lots of hard coded IPs in multiple places. Ergo puppet code per site.
![Page 30: Test Driven Infrastructure Development](https://reader037.vdocument.in/reader037/viewer/2022102815/5562b7e3d8b42a09618b46c8/html5/thumbnails/30.jpg)
• Hard coded IPs in 10 places
• role::oy_lb
Unfortunate reality:
Thursday, 14 March 13
So, despite our best efforts, our puppet code was SHIIIIT.Exported resources IS NOT a good fit for non-trivial things (like generating load balancer configs). Ergo lots of hard coded IPs in multiple places. Ergo puppet code per site.
![Page 31: Test Driven Infrastructure Development](https://reader037.vdocument.in/reader037/viewer/2022102815/5562b7e3d8b42a09618b46c8/html5/thumbnails/31.jpg)
• Hard coded IPs in 10 places
• role::oy_lb
• hiera data split by domain (colo)
Unfortunate reality:
Thursday, 14 March 13
So, despite our best efforts, our puppet code was SHIIIIT.Exported resources IS NOT a good fit for non-trivial things (like generating load balancer configs). Ergo lots of hard coded IPs in multiple places. Ergo puppet code per site.
![Page 32: Test Driven Infrastructure Development](https://reader037.vdocument.in/reader037/viewer/2022102815/5562b7e3d8b42a09618b46c8/html5/thumbnails/32.jpg)
• Hard coded IPs in 10 places
• role::oy_lb
• hiera data split by domain (colo)
• mco puppet
Unfortunate reality:
Thursday, 14 March 13
So, despite our best efforts, our puppet code was SHIIIIT.Exported resources IS NOT a good fit for non-trivial things (like generating load balancer configs). Ergo lots of hard coded IPs in multiple places. Ergo puppet code per site.
![Page 33: Test Driven Infrastructure Development](https://reader037.vdocument.in/reader037/viewer/2022102815/5562b7e3d8b42a09618b46c8/html5/thumbnails/33.jpg)
• Hard coded IPs in 10 places
• role::oy_lb
• hiera data split by domain (colo)
• mco puppet
• 4 weeks per app per environment
Unfortunate reality:
Thursday, 14 March 13
So, despite our best efforts, our puppet code was SHIIIIT.Exported resources IS NOT a good fit for non-trivial things (like generating load balancer configs). Ergo lots of hard coded IPs in multiple places. Ergo puppet code per site.
![Page 34: Test Driven Infrastructure Development](https://reader037.vdocument.in/reader037/viewer/2022102815/5562b7e3d8b42a09618b46c8/html5/thumbnails/34.jpg)
The state of the art
Thursday, 14 March 13
![Page 35: Test Driven Infrastructure Development](https://reader037.vdocument.in/reader037/viewer/2022102815/5562b7e3d8b42a09618b46c8/html5/thumbnails/35.jpg)
The state of the art
• It’s certainly in a state
Thursday, 14 March 13
Nobody does automatic runsPuppet becomes an auditing tool (automatic noop runs + reports)
![Page 36: Test Driven Infrastructure Development](https://reader037.vdocument.in/reader037/viewer/2022102815/5562b7e3d8b42a09618b46c8/html5/thumbnails/36.jpg)
The state of the art
• It’s certainly in a state
• Automatic runs dangerous
Thursday, 14 March 13
Nobody does automatic runsPuppet becomes an auditing tool (automatic noop runs + reports)
![Page 37: Test Driven Infrastructure Development](https://reader037.vdocument.in/reader037/viewer/2022102815/5562b7e3d8b42a09618b46c8/html5/thumbnails/37.jpg)
The state of the art
• It’s certainly in a state
• Automatic runs dangerous
• cron --noop runs
Thursday, 14 March 13
Nobody does automatic runsPuppet becomes an auditing tool (automatic noop runs + reports)
![Page 38: Test Driven Infrastructure Development](https://reader037.vdocument.in/reader037/viewer/2022102815/5562b7e3d8b42a09618b46c8/html5/thumbnails/38.jpg)
The state of the art
• It’s certainly in a state
• Automatic runs dangerous
• cron --noop runs
• puppet becomes an auditing system
Thursday, 14 March 13
Nobody does automatic runsPuppet becomes an auditing tool (automatic noop runs + reports)
![Page 39: Test Driven Infrastructure Development](https://reader037.vdocument.in/reader037/viewer/2022102815/5562b7e3d8b42a09618b46c8/html5/thumbnails/39.jpg)
The state of the art
• It’s certainly in a state
• Automatic runs dangerous
• cron --noop runs
• puppet becomes an auditing system
• This isn’t what I signed up for!
Thursday, 14 March 13
Nobody does automatic runsPuppet becomes an auditing tool (automatic noop runs + reports)
![Page 40: Test Driven Infrastructure Development](https://reader037.vdocument.in/reader037/viewer/2022102815/5562b7e3d8b42a09618b46c8/html5/thumbnails/40.jpg)
Business says no!
Thursday, 14 March 13
![Page 41: Test Driven Infrastructure Development](https://reader037.vdocument.in/reader037/viewer/2022102815/5562b7e3d8b42a09618b46c8/html5/thumbnails/41.jpg)
Business says no!
• Launching new products has a long lead time
• This is unhelpful if your company is trying to branch out into new markets
Thursday, 14 March 13
![Page 42: Test Driven Infrastructure Development](https://reader037.vdocument.in/reader037/viewer/2022102815/5562b7e3d8b42a09618b46c8/html5/thumbnails/42.jpg)
Business says no!
• Launching new products has a long lead time
• This is unhelpful if your company is trying to branch out into new markets
• CI / stage environments unlike prod
• Issues when new functionality goes live
• Developers think you’re incompetent
Thursday, 14 March 13
![Page 43: Test Driven Infrastructure Development](https://reader037.vdocument.in/reader037/viewer/2022102815/5562b7e3d8b42a09618b46c8/html5/thumbnails/43.jpg)
What is wrong with this picture?
Thursday, 14 March 13
You just don’t know the answer to any of these questions in any reliable way...But, generally, the answers are NO, YES, NO, NO
![Page 44: Test Driven Infrastructure Development](https://reader037.vdocument.in/reader037/viewer/2022102815/5562b7e3d8b42a09618b46c8/html5/thumbnails/44.jpg)
What is wrong with this picture?•Did you run it everywhere?
Thursday, 14 March 13
You just don’t know the answer to any of these questions in any reliable way...But, generally, the answers are NO, YES, NO, NO
![Page 45: Test Driven Infrastructure Development](https://reader037.vdocument.in/reader037/viewer/2022102815/5562b7e3d8b42a09618b46c8/html5/thumbnails/45.jpg)
What is wrong with this picture?•Did you run it everywhere?
•Does it affect anything you’re not expecting?
Thursday, 14 March 13
You just don’t know the answer to any of these questions in any reliable way...But, generally, the answers are NO, YES, NO, NO
![Page 46: Test Driven Infrastructure Development](https://reader037.vdocument.in/reader037/viewer/2022102815/5562b7e3d8b42a09618b46c8/html5/thumbnails/46.jpg)
What is wrong with this picture?•Did you run it everywhere?
•Does it affect anything you’re not expecting?
•Can you rebuild cleanly?
Thursday, 14 March 13
You just don’t know the answer to any of these questions in any reliable way...But, generally, the answers are NO, YES, NO, NO
![Page 47: Test Driven Infrastructure Development](https://reader037.vdocument.in/reader037/viewer/2022102815/5562b7e3d8b42a09618b46c8/html5/thumbnails/47.jpg)
What is wrong with this picture?•Did you run it everywhere?
•Does it affect anything you’re not expecting?
•Can you rebuild cleanly?
•Does the code even make things reflect current state?
Thursday, 14 March 13
You just don’t know the answer to any of these questions in any reliable way...But, generally, the answers are NO, YES, NO, NO
![Page 48: Test Driven Infrastructure Development](https://reader037.vdocument.in/reader037/viewer/2022102815/5562b7e3d8b42a09618b46c8/html5/thumbnails/48.jpg)
‘We use puppet’
Thursday, 14 March 13
Hint - you don’t!
![Page 49: Test Driven Infrastructure Development](https://reader037.vdocument.in/reader037/viewer/2022102815/5562b7e3d8b42a09618b46c8/html5/thumbnails/49.jpg)
‘We use puppet’
• Means nothing
Thursday, 14 March 13
Hint - you don’t!
![Page 50: Test Driven Infrastructure Development](https://reader037.vdocument.in/reader037/viewer/2022102815/5562b7e3d8b42a09618b46c8/html5/thumbnails/50.jpg)
‘We use puppet’
• Means nothing
• State of your system is the sum of all changes
Thursday, 14 March 13
Hint - you don’t!
![Page 51: Test Driven Infrastructure Development](https://reader037.vdocument.in/reader037/viewer/2022102815/5562b7e3d8b42a09618b46c8/html5/thumbnails/51.jpg)
‘We use puppet’
• Means nothing
• State of your system is the sum of all changes
• How do you know your code can rebuild things?
Thursday, 14 March 13
Hint - you don’t!
![Page 52: Test Driven Infrastructure Development](https://reader037.vdocument.in/reader037/viewer/2022102815/5562b7e3d8b42a09618b46c8/html5/thumbnails/52.jpg)
It’s all mierda
Thursday, 14 March 13
We need to grow up, and raise the level of the conversation..
![Page 53: Test Driven Infrastructure Development](https://reader037.vdocument.in/reader037/viewer/2022102815/5562b7e3d8b42a09618b46c8/html5/thumbnails/53.jpg)
It’s all mierda
•Development communities are 10 years ahead
Thursday, 14 March 13
We need to grow up, and raise the level of the conversation..
![Page 54: Test Driven Infrastructure Development](https://reader037.vdocument.in/reader037/viewer/2022102815/5562b7e3d8b42a09618b46c8/html5/thumbnails/54.jpg)
It’s all mierda
•Development communities are 10 years ahead
•We don’t integration test
• (repeatably)
Thursday, 14 March 13
We need to grow up, and raise the level of the conversation..
![Page 55: Test Driven Infrastructure Development](https://reader037.vdocument.in/reader037/viewer/2022102815/5562b7e3d8b42a09618b46c8/html5/thumbnails/55.jpg)
It’s all mierda
•Development communities are 10 years ahead
•We don’t integration test
• (repeatably)
•We can’t build / rebuild
• (reliably)
Thursday, 14 March 13
We need to grow up, and raise the level of the conversation..
![Page 56: Test Driven Infrastructure Development](https://reader037.vdocument.in/reader037/viewer/2022102815/5562b7e3d8b42a09618b46c8/html5/thumbnails/56.jpg)
Infra is hard
Thursday, 14 March 13
Sure - it’s much much harder to get a standalone testable system in infra than it is in development.
![Page 57: Test Driven Infrastructure Development](https://reader037.vdocument.in/reader037/viewer/2022102815/5562b7e3d8b42a09618b46c8/html5/thumbnails/57.jpg)
Infra is hard
• Infrastructure is inherently more complex
Thursday, 14 March 13
Sure - it’s much much harder to get a standalone testable system in infra than it is in development.
![Page 58: Test Driven Infrastructure Development](https://reader037.vdocument.in/reader037/viewer/2022102815/5562b7e3d8b42a09618b46c8/html5/thumbnails/58.jpg)
Infra is hard
• Infrastructure is inherently more complex
• Less control
Thursday, 14 March 13
Sure - it’s much much harder to get a standalone testable system in infra than it is in development.
![Page 59: Test Driven Infrastructure Development](https://reader037.vdocument.in/reader037/viewer/2022102815/5562b7e3d8b42a09618b46c8/html5/thumbnails/59.jpg)
Infra is hard
• Infrastructure is inherently more complex
• Less control
• More moving parts
Thursday, 14 March 13
Sure - it’s much much harder to get a standalone testable system in infra than it is in development.
![Page 60: Test Driven Infrastructure Development](https://reader037.vdocument.in/reader037/viewer/2022102815/5562b7e3d8b42a09618b46c8/html5/thumbnails/60.jpg)
Infra is hard
• Infrastructure is inherently more complex
• Less control
• More moving parts
• ‘End to end’ testing
Thursday, 14 March 13
Sure - it’s much much harder to get a standalone testable system in infra than it is in development.
![Page 61: Test Driven Infrastructure Development](https://reader037.vdocument.in/reader037/viewer/2022102815/5562b7e3d8b42a09618b46c8/html5/thumbnails/61.jpg)
Infra is hard
• Infrastructure is inherently more complex
• Less control
• More moving parts
• ‘End to end’ testing
• Persistent data
Thursday, 14 March 13
Sure - it’s much much harder to get a standalone testable system in infra than it is in development.
![Page 62: Test Driven Infrastructure Development](https://reader037.vdocument.in/reader037/viewer/2022102815/5562b7e3d8b42a09618b46c8/html5/thumbnails/62.jpg)
No excuses:Scientific method
Thursday, 14 March 13
I do not consider this an excuse to abandon sanity.
![Page 63: Test Driven Infrastructure Development](https://reader037.vdocument.in/reader037/viewer/2022102815/5562b7e3d8b42a09618b46c8/html5/thumbnails/63.jpg)
The solution?
Thursday, 14 March 13
![Page 64: Test Driven Infrastructure Development](https://reader037.vdocument.in/reader037/viewer/2022102815/5562b7e3d8b42a09618b46c8/html5/thumbnails/64.jpg)
The solution?
• Re-provision everything in tests
• N.B. Not perfect (but better!)
Thursday, 14 March 13
![Page 65: Test Driven Infrastructure Development](https://reader037.vdocument.in/reader037/viewer/2022102815/5562b7e3d8b42a09618b46c8/html5/thumbnails/65.jpg)
The solution?
• Re-provision everything in tests
• N.B. Not perfect (but better!)
Thursday, 14 March 13
![Page 66: Test Driven Infrastructure Development](https://reader037.vdocument.in/reader037/viewer/2022102815/5562b7e3d8b42a09618b46c8/html5/thumbnails/66.jpg)
The solution?
• Re-provision everything in tests
• N.B. Not perfect (but better!)
• Proper software engineering
• Unit and integration tests
• Build pipeline + promotion
Thursday, 14 March 13
![Page 67: Test Driven Infrastructure Development](https://reader037.vdocument.in/reader037/viewer/2022102815/5562b7e3d8b42a09618b46c8/html5/thumbnails/67.jpg)
Openstack
• Our tests spinning up 12 machines => VMs
Thursday, 14 March 13
So, we should use openstack, right? As of December, when we looked - 2 networks max, inflexible. lvs not possible.
![Page 68: Test Driven Infrastructure Development](https://reader037.vdocument.in/reader037/viewer/2022102815/5562b7e3d8b42a09618b46c8/html5/thumbnails/68.jpg)
Openstack
• Our tests spinning up 12 machines => VMs
• Openstack going to be awesome, right now:
Thursday, 14 March 13
So, we should use openstack, right? As of December, when we looked - 2 networks max, inflexible. lvs not possible.
![Page 69: Test Driven Infrastructure Development](https://reader037.vdocument.in/reader037/viewer/2022102815/5562b7e3d8b42a09618b46c8/html5/thumbnails/69.jpg)
Openstack
• Our tests spinning up 12 machines => VMs
• Openstack going to be awesome, right now:
• Networking sucks
Thursday, 14 March 13
So, we should use openstack, right? As of December, when we looked - 2 networks max, inflexible. lvs not possible.
![Page 70: Test Driven Infrastructure Development](https://reader037.vdocument.in/reader037/viewer/2022102815/5562b7e3d8b42a09618b46c8/html5/thumbnails/70.jpg)
Openstack
• Our tests spinning up 12 machines => VMs
• Openstack going to be awesome, right now:
• Networking sucks
• Load balancing is a shambles
Thursday, 14 March 13
So, we should use openstack, right? As of December, when we looked - 2 networks max, inflexible. lvs not possible.
![Page 71: Test Driven Infrastructure Development](https://reader037.vdocument.in/reader037/viewer/2022102815/5562b7e3d8b42a09618b46c8/html5/thumbnails/71.jpg)
Openstack
• Our tests spinning up 12 machines => VMs
• Openstack going to be awesome, right now:
• Networking sucks
• Load balancing is a shambles
• lvs / vlans / metal / bonding - nope
Thursday, 14 March 13
So, we should use openstack, right? As of December, when we looked - 2 networks max, inflexible. lvs not possible.
![Page 72: Test Driven Infrastructure Development](https://reader037.vdocument.in/reader037/viewer/2022102815/5562b7e3d8b42a09618b46c8/html5/thumbnails/72.jpg)
My desires:
Thursday, 14 March 13
![Page 73: Test Driven Infrastructure Development](https://reader037.vdocument.in/reader037/viewer/2022102815/5562b7e3d8b42a09618b46c8/html5/thumbnails/73.jpg)
My desires:• Reuse as much code as possible! (e.g. load
balancers)
Thursday, 14 March 13
![Page 74: Test Driven Infrastructure Development](https://reader037.vdocument.in/reader037/viewer/2022102815/5562b7e3d8b42a09618b46c8/html5/thumbnails/74.jpg)
My desires:• Reuse as much code as possible! (e.g. load
balancers)
• No per colo/environment puppet code
Thursday, 14 March 13
![Page 75: Test Driven Infrastructure Development](https://reader037.vdocument.in/reader037/viewer/2022102815/5562b7e3d8b42a09618b46c8/html5/thumbnails/75.jpg)
My desires:• Reuse as much code as possible! (e.g. load
balancers)
• No per colo/environment puppet code
• No IPs anywhere
Thursday, 14 March 13
![Page 76: Test Driven Infrastructure Development](https://reader037.vdocument.in/reader037/viewer/2022102815/5562b7e3d8b42a09618b46c8/html5/thumbnails/76.jpg)
My desires:• Reuse as much code as possible! (e.g. load
balancers)
• No per colo/environment puppet code
• No IPs anywhere
• ‘DRY’
Thursday, 14 March 13
![Page 77: Test Driven Infrastructure Development](https://reader037.vdocument.in/reader037/viewer/2022102815/5562b7e3d8b42a09618b46c8/html5/thumbnails/77.jpg)
My desires:• Reuse as much code as possible! (e.g. load
balancers)
• No per colo/environment puppet code
• No IPs anywhere
• ‘DRY’
• CI pipeline to promote to production
Thursday, 14 March 13
![Page 78: Test Driven Infrastructure Development](https://reader037.vdocument.in/reader037/viewer/2022102815/5562b7e3d8b42a09618b46c8/html5/thumbnails/78.jpg)
My desires:• Reuse as much code as possible! (e.g. load
balancers)
• No per colo/environment puppet code
• No IPs anywhere
• ‘DRY’
• CI pipeline to promote to production
• 1 puppet run from provisioned to working
Thursday, 14 March 13
![Page 79: Test Driven Infrastructure Development](https://reader037.vdocument.in/reader037/viewer/2022102815/5562b7e3d8b42a09618b46c8/html5/thumbnails/79.jpg)
My desires:• Reuse as much code as possible! (e.g. load
balancers)
• No per colo/environment puppet code
• No IPs anywhere
• ‘DRY’
• CI pipeline to promote to production
• 1 puppet run from provisioned to working
• Repeatable and testable!
Thursday, 14 March 13
![Page 80: Test Driven Infrastructure Development](https://reader037.vdocument.in/reader037/viewer/2022102815/5562b7e3d8b42a09618b46c8/html5/thumbnails/80.jpg)
Orc
Thursday, 14 March 13
![Page 81: Test Driven Infrastructure Development](https://reader037.vdocument.in/reader037/viewer/2022102815/5562b7e3d8b42a09618b46c8/html5/thumbnails/81.jpg)
Orc• Continuous (zero downtime) deployment
Thursday, 14 March 13
![Page 82: Test Driven Infrastructure Development](https://reader037.vdocument.in/reader037/viewer/2022102815/5562b7e3d8b42a09618b46c8/html5/thumbnails/82.jpg)
Orc• Continuous (zero downtime) deployment
• Development / infrastructure application contract
Thursday, 14 March 13
![Page 83: Test Driven Infrastructure Development](https://reader037.vdocument.in/reader037/viewer/2022102815/5562b7e3d8b42a09618b46c8/html5/thumbnails/83.jpg)
Orc• Continuous (zero downtime) deployment
• Development / infrastructure application contract
• Model driven
Thursday, 14 March 13
![Page 84: Test Driven Infrastructure Development](https://reader037.vdocument.in/reader037/viewer/2022102815/5562b7e3d8b42a09618b46c8/html5/thumbnails/84.jpg)
Orc• Continuous (zero downtime) deployment
• Development / infrastructure application contract
• Model driven
• https://github.com/youdevise/orc/
Thursday, 14 March 13
![Page 85: Test Driven Infrastructure Development](https://reader037.vdocument.in/reader037/viewer/2022102815/5562b7e3d8b42a09618b46c8/html5/thumbnails/85.jpg)
Puppetroll
Thursday, 14 March 13
![Page 86: Test Driven Infrastructure Development](https://reader037.vdocument.in/reader037/viewer/2022102815/5562b7e3d8b42a09618b46c8/html5/thumbnails/86.jpg)
Puppetroll
• Rolls out a consistent sha1 from the puppetmaster to an entire environment
Thursday, 14 March 13
![Page 87: Test Driven Infrastructure Development](https://reader037.vdocument.in/reader037/viewer/2022102815/5562b7e3d8b42a09618b46c8/html5/thumbnails/87.jpg)
Puppetroll
• Rolls out a consistent sha1 from the puppetmaster to an entire environment
• Fails if any puppet run fails
Thursday, 14 March 13
![Page 88: Test Driven Infrastructure Development](https://reader037.vdocument.in/reader037/viewer/2022102815/5562b7e3d8b42a09618b46c8/html5/thumbnails/88.jpg)
Puppetroll
• Rolls out a consistent sha1 from the puppetmaster to an entire environment
• Fails if any puppet run fails
• https://github.com/youdevise/puppetroll
Thursday, 14 March 13
![Page 89: Test Driven Infrastructure Development](https://reader037.vdocument.in/reader037/viewer/2022102815/5562b7e3d8b42a09618b46c8/html5/thumbnails/89.jpg)
Provisioning tools
Thursday, 14 March 13
![Page 90: Test Driven Infrastructure Development](https://reader037.vdocument.in/reader037/viewer/2022102815/5562b7e3d8b42a09618b46c8/html5/thumbnails/90.jpg)
Provisioning tools
• debootstrap custom gold images
Thursday, 14 March 13
![Page 91: Test Driven Infrastructure Development](https://reader037.vdocument.in/reader037/viewer/2022102815/5562b7e3d8b42a09618b46c8/html5/thumbnails/91.jpg)
Provisioning tools
• debootstrap custom gold images
• mcollective ‘computenode’ agent for kvm
Thursday, 14 March 13
![Page 92: Test Driven Infrastructure Development](https://reader037.vdocument.in/reader037/viewer/2022102815/5562b7e3d8b42a09618b46c8/html5/thumbnails/92.jpg)
Provisioning tools
• debootstrap custom gold images
• mcollective ‘computenode’ agent for kvm
• ‘provision me a machine called X, on networks Y and Z’
Thursday, 14 March 13
![Page 93: Test Driven Infrastructure Development](https://reader037.vdocument.in/reader037/viewer/2022102815/5562b7e3d8b42a09618b46c8/html5/thumbnails/93.jpg)
Provisioning tools
• debootstrap custom gold images
• mcollective ‘computenode’ agent for kvm
• ‘provision me a machine called X, on networks Y and Z’
• Dynamic IP allocation (dnsmasq locally, DDNS for real)
Thursday, 14 March 13
![Page 94: Test Driven Infrastructure Development](https://reader037.vdocument.in/reader037/viewer/2022102815/5562b7e3d8b42a09618b46c8/html5/thumbnails/94.jpg)
stacks
Thursday, 14 March 13
![Page 95: Test Driven Infrastructure Development](https://reader037.vdocument.in/reader037/viewer/2022102815/5562b7e3d8b42a09618b46c8/html5/thumbnails/95.jpg)
stacks
• Model driven deployment
Thursday, 14 March 13
![Page 96: Test Driven Infrastructure Development](https://reader037.vdocument.in/reader037/viewer/2022102815/5562b7e3d8b42a09618b46c8/html5/thumbnails/96.jpg)
stacks
• Model driven deployment
• DSL for describing groups of systems + dependencies
Thursday, 14 March 13
![Page 97: Test Driven Infrastructure Development](https://reader037.vdocument.in/reader037/viewer/2022102815/5562b7e3d8b42a09618b46c8/html5/thumbnails/97.jpg)
stacks
• Model driven deployment
• DSL for describing groups of systems + dependencies
• rake tasks to provision / test / clean up stack + deps
Thursday, 14 March 13
![Page 98: Test Driven Infrastructure Development](https://reader037.vdocument.in/reader037/viewer/2022102815/5562b7e3d8b42a09618b46c8/html5/thumbnails/98.jpg)
stacks
• Model driven deployment
• DSL for describing groups of systems + dependencies
• rake tasks to provision / test / clean up stack + deps
• Can provision a full environment, run E2E tests, tear it down - in CI.
Thursday, 14 March 13
![Page 99: Test Driven Infrastructure Development](https://reader037.vdocument.in/reader037/viewer/2022102815/5562b7e3d8b42a09618b46c8/html5/thumbnails/99.jpg)
Thursday, 14 March 13
![Page 100: Test Driven Infrastructure Development](https://reader037.vdocument.in/reader037/viewer/2022102815/5562b7e3d8b42a09618b46c8/html5/thumbnails/100.jpg)
Thursday, 14 March 13
![Page 101: Test Driven Infrastructure Development](https://reader037.vdocument.in/reader037/viewer/2022102815/5562b7e3d8b42a09618b46c8/html5/thumbnails/101.jpg)
I want to hack on load balancers
= 4 new, independent machines
Thursday, 14 March 13
![Page 102: Test Driven Infrastructure Development](https://reader037.vdocument.in/reader037/viewer/2022102815/5562b7e3d8b42a09618b46c8/html5/thumbnails/102.jpg)
How it works?
Thursday, 14 March 13
![Page 103: Test Driven Infrastructure Development](https://reader037.vdocument.in/reader037/viewer/2022102815/5562b7e3d8b42a09618b46c8/html5/thumbnails/103.jpg)
How it works?
• DSL creates model of systems
Thursday, 14 March 13
![Page 104: Test Driven Infrastructure Development](https://reader037.vdocument.in/reader037/viewer/2022102815/5562b7e3d8b42a09618b46c8/html5/thumbnails/104.jpg)
How it works?
• DSL creates model of systems
• rake task ‘launch’:
Thursday, 14 March 13
![Page 105: Test Driven Infrastructure Development](https://reader037.vdocument.in/reader037/viewer/2022102815/5562b7e3d8b42a09618b46c8/html5/thumbnails/105.jpg)
How it works?
• DSL creates model of systems
• rake task ‘launch’:
• mco provisions boxes on compute nodes
Thursday, 14 March 13
![Page 106: Test Driven Infrastructure Development](https://reader037.vdocument.in/reader037/viewer/2022102815/5562b7e3d8b42a09618b46c8/html5/thumbnails/106.jpg)
How it works?
• DSL creates model of systems
• rake task ‘launch’:
• mco provisions boxes on compute nodes
• each box runs puppet --waitforcert
Thursday, 14 March 13
![Page 107: Test Driven Infrastructure Development](https://reader037.vdocument.in/reader037/viewer/2022102815/5562b7e3d8b42a09618b46c8/html5/thumbnails/107.jpg)
How it works?
• DSL creates model of systems
• rake task ‘launch’:
• mco provisions boxes on compute nodes
• each box runs puppet --waitforcert
• mco signs cert
Thursday, 14 March 13
![Page 108: Test Driven Infrastructure Development](https://reader037.vdocument.in/reader037/viewer/2022102815/5562b7e3d8b42a09618b46c8/html5/thumbnails/108.jpg)
How it works?
• DSL creates model of systems
• rake task ‘launch’:
• mco provisions boxes on compute nodes
• each box runs puppet --waitforcert
• mco signs cert
• puppet runs for each box
Thursday, 14 March 13
![Page 109: Test Driven Infrastructure Development](https://reader037.vdocument.in/reader037/viewer/2022102815/5562b7e3d8b42a09618b46c8/html5/thumbnails/109.jpg)
mco computenode
Thursday, 14 March 13
![Page 110: Test Driven Infrastructure Development](https://reader037.vdocument.in/reader037/viewer/2022102815/5562b7e3d8b42a09618b46c8/html5/thumbnails/110.jpg)
Puppetmaster
Thursday, 14 March 13
![Page 111: Test Driven Infrastructure Development](https://reader037.vdocument.in/reader037/viewer/2022102815/5562b7e3d8b42a09618b46c8/html5/thumbnails/111.jpg)
Puppetmaster
• Uses the same model
Thursday, 14 March 13
![Page 112: Test Driven Infrastructure Development](https://reader037.vdocument.in/reader037/viewer/2022102815/5562b7e3d8b42a09618b46c8/html5/thumbnails/112.jpg)
Puppetmaster
• Uses the same model
• Generates an ENC for each node
Thursday, 14 March 13
![Page 113: Test Driven Infrastructure Development](https://reader037.vdocument.in/reader037/viewer/2022102815/5562b7e3d8b42a09618b46c8/html5/thumbnails/113.jpg)
Puppetmaster
• Uses the same model
• Generates an ENC for each node
• Puppet code:
Thursday, 14 March 13
![Page 114: Test Driven Infrastructure Development](https://reader037.vdocument.in/reader037/viewer/2022102815/5562b7e3d8b42a09618b46c8/html5/thumbnails/114.jpg)
Puppetmaster
• Uses the same model
• Generates an ENC for each node
• Puppet code:
• Just installs things / starts services
Thursday, 14 March 13
![Page 115: Test Driven Infrastructure Development](https://reader037.vdocument.in/reader037/viewer/2022102815/5562b7e3d8b42a09618b46c8/html5/thumbnails/115.jpg)
Puppetmaster
• Uses the same model
• Generates an ENC for each node
• Puppet code:
• Just installs things / starts services
• I.E. what it’s good at!
Thursday, 14 March 13
![Page 116: Test Driven Infrastructure Development](https://reader037.vdocument.in/reader037/viewer/2022102815/5562b7e3d8b42a09618b46c8/html5/thumbnails/116.jpg)
External node classifier
Thursday, 14 March 13
![Page 117: Test Driven Infrastructure Development](https://reader037.vdocument.in/reader037/viewer/2022102815/5562b7e3d8b42a09618b46c8/html5/thumbnails/117.jpg)
Putting it together
Thursday, 14 March 13
So, what do we have? Well - everything I showed you already...Building proxy server layer (by refactoring puppet code) right now. Databases to follow!
![Page 118: Test Driven Infrastructure Development](https://reader037.vdocument.in/reader037/viewer/2022102815/5562b7e3d8b42a09618b46c8/html5/thumbnails/118.jpg)
Putting it together
• Still ongoing - live production apps ETA two weeks.
Thursday, 14 March 13
So, what do we have? Well - everything I showed you already...Building proxy server layer (by refactoring puppet code) right now. Databases to follow!
![Page 119: Test Driven Infrastructure Development](https://reader037.vdocument.in/reader037/viewer/2022102815/5562b7e3d8b42a09618b46c8/html5/thumbnails/119.jpg)
Putting it together
• Still ongoing - live production apps ETA two weeks.
• Still haven’t solved re-provisioning problem for live environments!
Thursday, 14 March 13
So, what do we have? Well - everything I showed you already...Building proxy server layer (by refactoring puppet code) right now. Databases to follow!
![Page 120: Test Driven Infrastructure Development](https://reader037.vdocument.in/reader037/viewer/2022102815/5562b7e3d8b42a09618b46c8/html5/thumbnails/120.jpg)
Putting it together
• Still ongoing - live production apps ETA two weeks.
• Still haven’t solved re-provisioning problem for live environments!
• Do have repeatable and testable / tested infrastructure building in CI!
Thursday, 14 March 13
So, what do we have? Well - everything I showed you already...Building proxy server layer (by refactoring puppet code) right now. Databases to follow!
![Page 121: Test Driven Infrastructure Development](https://reader037.vdocument.in/reader037/viewer/2022102815/5562b7e3d8b42a09618b46c8/html5/thumbnails/121.jpg)
Thursday, 14 March 13
![Page 122: Test Driven Infrastructure Development](https://reader037.vdocument.in/reader037/viewer/2022102815/5562b7e3d8b42a09618b46c8/html5/thumbnails/122.jpg)
Thursday, 14 March 13
The top table is our test overview - we have two types of tests, those which are for a specific machine (i.e. a VM) and those which are for a virtual service (backed by multiple machines)‘behaves like’ is an rspec thing we haven’t overridden.For each machine, we test that it’s pingable, then run every nrpe (nagios) agent and check them - if all the things nagios monitors are OK, machine is OK.
![Page 123: Test Driven Infrastructure Development](https://reader037.vdocument.in/reader037/viewer/2022102815/5562b7e3d8b42a09618b46c8/html5/thumbnails/123.jpg)
In the (near) future?
Thursday, 14 March 13
![Page 124: Test Driven Infrastructure Development](https://reader037.vdocument.in/reader037/viewer/2022102815/5562b7e3d8b42a09618b46c8/html5/thumbnails/124.jpg)
In the (near) future?
• Live application stack in production
Thursday, 14 March 13
![Page 125: Test Driven Infrastructure Development](https://reader037.vdocument.in/reader037/viewer/2022102815/5562b7e3d8b42a09618b46c8/html5/thumbnails/125.jpg)
In the (near) future?
• Live application stack in production
• Automated ‘promotion’ of good changes to production
Thursday, 14 March 13
![Page 126: Test Driven Infrastructure Development](https://reader037.vdocument.in/reader037/viewer/2022102815/5562b7e3d8b42a09618b46c8/html5/thumbnails/126.jpg)
In the (near) future?
• Live application stack in production
• Automated ‘promotion’ of good changes to production
• Integrated environment support for dev stacks on dev branches/environments
Thursday, 14 March 13
![Page 127: Test Driven Infrastructure Development](https://reader037.vdocument.in/reader037/viewer/2022102815/5562b7e3d8b42a09618b46c8/html5/thumbnails/127.jpg)
In the (near) future?
• Live application stack in production
• Automated ‘promotion’ of good changes to production
• Integrated environment support for dev stacks on dev branches/environments
• Open source all the things!
Thursday, 14 March 13
![Page 128: Test Driven Infrastructure Development](https://reader037.vdocument.in/reader037/viewer/2022102815/5562b7e3d8b42a09618b46c8/html5/thumbnails/128.jpg)
Thanks!
Thursday, 14 March 13
![Page 129: Test Driven Infrastructure Development](https://reader037.vdocument.in/reader037/viewer/2022102815/5562b7e3d8b42a09618b46c8/html5/thumbnails/129.jpg)
Thanks!
• puppet is an awesome tool.
• It doesn’t solve higher level system modeling problems
• It shouldn’t try to!
Thursday, 14 March 13
![Page 130: Test Driven Infrastructure Development](https://reader037.vdocument.in/reader037/viewer/2022102815/5562b7e3d8b42a09618b46c8/html5/thumbnails/130.jpg)
Thanks!
• puppet is an awesome tool.
• It doesn’t solve higher level system modeling problems
• It shouldn’t try to!
• sysadmins need to level up
• It’s not done till you can test it still works
Thursday, 14 March 13
![Page 131: Test Driven Infrastructure Development](https://reader037.vdocument.in/reader037/viewer/2022102815/5562b7e3d8b42a09618b46c8/html5/thumbnails/131.jpg)
Photo Credits• Escher's "Relativity" in LEGO - Andrew Lipson (http://www.andrewlipson.com)
• Manure - Flickr - chesbayprogram
• Provisions - Flickr - quinn.anya
• Stacked - Flickr - andrewrennie
• Dilbert - Flickr - osde-info
• Stacking wood - fickr - arthuserea
• Square wheels - Flickr - vrogy
• Puppets - Flickr - SkipSteuart
• Light bulb - Flickr - bazik
• This-is-not-art - Wikimedia commons - Loran Davis
• Danger of death - Flickr - zigazou76
• Bob the Builder - Flickr - jamesclay
• Swiss roll - Flickr - add1sun
• Orc - Flickr - photo_munki
• Danger! Danger - Flickr - donsolo
• Cow of the future - Flickr - thewamphyri
• SCIENCE - Flickr - chasblackman
Thursday, 14 March 13
![Page 132: Test Driven Infrastructure Development](https://reader037.vdocument.in/reader037/viewer/2022102815/5562b7e3d8b42a09618b46c8/html5/thumbnails/132.jpg)
Links!
• http://github.com/youdevise
• http://github.com/bobtfish
• https://devblog.timgroup.com/
• (Yes, we are hiring)
Thursday, 14 March 13