building a better mousetrap for the cloud
DESCRIPTION
TRANSCRIPT
![Page 1: Building a Better Mousetrap for the Cloud](https://reader038.vdocument.in/reader038/viewer/2022102921/5456e139b1af9fba5d8b4742/html5/thumbnails/1.jpg)
Rube Goldberg Architecture
Ezra Zygmuntowicz
Building a Better Mousetrap for the
Cloud
![Page 2: Building a Better Mousetrap for the Cloud](https://reader038.vdocument.in/reader038/viewer/2022102921/5456e139b1af9fba5d8b4742/html5/thumbnails/2.jpg)
3 Pillars of a good Cloud Infrastructure
Automation
Command & Control
Scalable State Storage
![Page 3: Building a Better Mousetrap for the Cloud](https://reader038.vdocument.in/reader038/viewer/2022102921/5456e139b1af9fba5d8b4742/html5/thumbnails/3.jpg)
Automation:Chef
Command & Control:Nanite
Scalable State Storage:Redis
![Page 4: Building a Better Mousetrap for the Cloud](https://reader038.vdocument.in/reader038/viewer/2022102921/5456e139b1af9fba5d8b4742/html5/thumbnails/4.jpg)
Go big or go home...
![Page 5: Building a Better Mousetrap for the Cloud](https://reader038.vdocument.in/reader038/viewer/2022102921/5456e139b1af9fba5d8b4742/html5/thumbnails/5.jpg)
Automation: Chef
• Idempotent configuration management
• Embed-able and flexible
• Heavy lifting behind Solo/Flex
• No more “every server is a unique snowflake”
• Badass
![Page 6: Building a Better Mousetrap for the Cloud](https://reader038.vdocument.in/reader038/viewer/2022102921/5456e139b1af9fba5d8b4742/html5/thumbnails/6.jpg)
Basic Concepts:
Resources
Recipes
Providers
![Page 7: Building a Better Mousetrap for the Cloud](https://reader038.vdocument.in/reader038/viewer/2022102921/5456e139b1af9fba5d8b4742/html5/thumbnails/7.jpg)
Resources:
apt_packagebashcroncsh
directoryexecute
filegem_package
grouphttp_request
linkmount
packageperl
portage_packagepython
remote_directoryremote_file
routerubyscript
servicetemplate
user
![Page 8: Building a Better Mousetrap for the Cloud](https://reader038.vdocument.in/reader038/viewer/2022102921/5456e139b1af9fba5d8b4742/html5/thumbnails/8.jpg)
Recipes:
![Page 9: Building a Better Mousetrap for the Cloud](https://reader038.vdocument.in/reader038/viewer/2022102921/5456e139b1af9fba5d8b4742/html5/thumbnails/9.jpg)
Providers:
![Page 10: Building a Better Mousetrap for the Cloud](https://reader038.vdocument.in/reader038/viewer/2022102921/5456e139b1af9fba5d8b4742/html5/thumbnails/10.jpg)
Converging
• Recipes are loaded in specified order
• Resources are compiled into objects and stored in a ResourceCollection
• ResourceCollection is iterated and the right Provider for each Resource is invoked
• The Providers runs the specified action on each Resource
![Page 11: Building a Better Mousetrap for the Cloud](https://reader038.vdocument.in/reader038/viewer/2022102921/5456e139b1af9fba5d8b4742/html5/thumbnails/11.jpg)
Chef Solo
• chef-solo -r http://foo.com/recipes.tgz
• Downloads a tarball of recipes and runs them
• Dead simple to get started with
![Page 12: Building a Better Mousetrap for the Cloud](https://reader038.vdocument.in/reader038/viewer/2022102921/5456e139b1af9fba5d8b4742/html5/thumbnails/12.jpg)
Chef Server
• chef-client
• Communicates with the chef-server to get recipes and JSON data
• Allows for searching across other nodes attributes
• Uses open-id for node authentication
![Page 13: Building a Better Mousetrap for the Cloud](https://reader038.vdocument.in/reader038/viewer/2022102921/5456e139b1af9fba5d8b4742/html5/thumbnails/13.jpg)
Chef Server
![Page 14: Building a Better Mousetrap for the Cloud](https://reader038.vdocument.in/reader038/viewer/2022102921/5456e139b1af9fba5d8b4742/html5/thumbnails/14.jpg)
State Storage: Redis
persistent memcached on steroids
![Page 15: Building a Better Mousetrap for the Cloud](https://reader038.vdocument.in/reader038/viewer/2022102921/5456e139b1af9fba5d8b4742/html5/thumbnails/15.jpg)
Redis Features
• Asynchronous Persist to disk
• Horizontally scalable
• Values can have types
• STRING’s, LIST’s and SET’s
• Atomic Operations (push, pop, incr, decr, set intersection)
![Page 16: Building a Better Mousetrap for the Cloud](https://reader038.vdocument.in/reader038/viewer/2022102921/5456e139b1af9fba5d8b4742/html5/thumbnails/16.jpg)
Let’s see a demo
![Page 17: Building a Better Mousetrap for the Cloud](https://reader038.vdocument.in/reader038/viewer/2022102921/5456e139b1af9fba5d8b4742/html5/thumbnails/17.jpg)
Nanite
![Page 18: Building a Better Mousetrap for the Cloud](https://reader038.vdocument.in/reader038/viewer/2022102921/5456e139b1af9fba5d8b4742/html5/thumbnails/18.jpg)
Built around RabbitMQ
• Written in erlang, cluster-able, highly scalable, fast as hell.
• AMQP protocol provides many nice features
• Transient, Persistent and Transactional semantics
![Page 19: Building a Better Mousetrap for the Cloud](https://reader038.vdocument.in/reader038/viewer/2022102921/5456e139b1af9fba5d8b4742/html5/thumbnails/19.jpg)
<3 RabbitMQ
![Page 20: Building a Better Mousetrap for the Cloud](https://reader038.vdocument.in/reader038/viewer/2022102921/5456e139b1af9fba5d8b4742/html5/thumbnails/20.jpg)
Nanite agentsconsist of multiple Actors
![Page 21: Building a Better Mousetrap for the Cloud](https://reader038.vdocument.in/reader038/viewer/2022102921/5456e139b1af9fba5d8b4742/html5/thumbnails/21.jpg)
Nanite agents advertisetheir services and status
Feeds#crawladvertises:/feeds/crawl
Load average is advertised as default status
![Page 22: Building a Better Mousetrap for the Cloud](https://reader038.vdocument.in/reader038/viewer/2022102921/5456e139b1af9fba5d8b4742/html5/thumbnails/22.jpg)
Nanite Mappers
Track nanites and their advertised services and status
Can do dispatch based on a number of factors
Run inside your Merb or Rails app or as a separate service
State of all nanites is replicated across all mappers in memory *or* stored in Redis
![Page 23: Building a Better Mousetrap for the Cloud](https://reader038.vdocument.in/reader038/viewer/2022102921/5456e139b1af9fba5d8b4742/html5/thumbnails/23.jpg)
Multiple Dispatch Styles
![Page 24: Building a Better Mousetrap for the Cloud](https://reader038.vdocument.in/reader038/viewer/2022102921/5456e139b1af9fba5d8b4742/html5/thumbnails/24.jpg)
Least loaded dispatch and the fitness function
![Page 25: Building a Better Mousetrap for the Cloud](https://reader038.vdocument.in/reader038/viewer/2022102921/5456e139b1af9fba5d8b4742/html5/thumbnails/25.jpg)
Agents ping the mapper exchange every @ping_time seconds.
Mappers track the state of all nanites and remove them from
mapping if they haven’t reported in within a timeout
![Page 26: Building a Better Mousetrap for the Cloud](https://reader038.vdocument.in/reader038/viewer/2022102921/5456e139b1af9fba5d8b4742/html5/thumbnails/26.jpg)
Nanite gives us:
• Presence, we know when nanites are ready for requests or not.
• Self assembly, nanites can come and go and can run anywhere with zero configuration in the mappers.
• Dispatch based on load or any fitness function that suits your app
• Easily take advantage of cloud
![Page 27: Building a Better Mousetrap for the Cloud](https://reader038.vdocument.in/reader038/viewer/2022102921/5456e139b1af9fba5d8b4742/html5/thumbnails/27.jpg)
https://cloud.engineyard.com/ete.sh
Please download and run this shell script:
Nanite Demo
![Page 28: Building a Better Mousetrap for the Cloud](https://reader038.vdocument.in/reader038/viewer/2022102921/5456e139b1af9fba5d8b4742/html5/thumbnails/28.jpg)
Questions?