amazon swf and gordon
DESCRIPTION
Presentation about Amazon SWF and the Gordon Ruby framework by Jonathan Weiss at the Ruby Yser Group Berlin on May 3rd 2012.TRANSCRIPT
Amazon SWF Ruby User Group Berlin
Peritor GmbH
EC2 & Eucalyptus Cluster Management
§ Automatic Configuration
§ Self Healing Clusters
§ Auto Scaling Clusters
§ One Click Deployment
§ One Click Cluster Cloning
§ Monitoring & Alerting
§ Logs & Accounting
§ Firewalls & Backups
§ Access & Rights Management
2
Architecture
3
DB
Web���App
Architecture
4
DB
Web���App
EC2
S3
Euca
Architecture
5
DB
Web���App
EC2
S3
Euca
Agent
RabbitMQ
RabbitMQ
6
RabbitMQ
7 http://www.heavensend-design.com/happy/
Agent/Queue Requirements
§ Scalability We manage thousands of instances
§ Availability & Reliability We need to be available even if an AWS region fails
8
DJ?
9
Resque?
10
Hmmmmm….
11
12
SWF: Simple Workflow Service
From http://aws.amazon.com/swf/
§ Maintains application state
§ Tracks workflow executions and logs their progress
§ Holds and dispatches tasks
§ Controls which tasks each of your application hosts will be assigned to execute
13
SWF: Simple Workflow Service
“Workflows as a Service”
§ Hosted
§ Scalable § Consistent (uses DynamoDB for state)
14
SWF: Pricing
§ $0.0001 per workflow execution
§ Until a Workflow Execution completes: $0.000005 per workflow execution for every 24-hours
§ $0.000005 per retained workflow execution per day
+ traffic
15
SWF: Workflow
16
Start ���Instance
Observe���Start
Send Init JSON
Cleanup
Activity 1 Activity 2 Activity 3
Activity 4
SWF: Architecture
17 http://docs.amazonwebservices.com/amazonswf/latest/developerguide/swf-welcome.html?r=8856
SWF: Decider
§ Implements workflow logic
§ Gets history of events
§ Decides what to do next: § Call next activity
§ Abort workflow
§ Re-schedule
§ …
18
SWF: Worker
§ Implements activity logic
§ Gets input for current activity
§ Processes activity: § Complete
§ Fail
§ Timeout
§ Heartbeat
19
Separate Task Order/Graph From Task Implementation
20
Worker Decider
21
Gordon
Ruby framework to use SWF
§ Syntactic sugar for creation of deciders and activities
§ Daemons for processing tasks
§ Unicorn-style master/child handling of processes
22
Gordon: Decider
23
Gordon: Decider
24
Gordon: Activity
25
Gordon: Start Workflow
26
Gordon: Processing
27
Gordon: Processing
28
Gordon: config.decider.yml
29
DEMO
30
Available soon at http://github.com/scalarium/gordon
Under Apache 2 license
31
SWF: Simple Workflow Service
Many nice features
§ Retention period
§ Task lists
§ Timeouts § Event history
§ Abort/cancel/re-schedule
§ Delay
§ Mark & count
§ ...
32
We’re Hiring!
DevOps
§ Scalarium operations
§ Write and maintain Chef cookbooks
§ Improve and scale backend
§ Security
§ Help us support more OSs
§ Vagrant integration
33
Ruby Developers
§ Rails frontend
§ Ruby backend / agents
§ Chef
§ Scaling evented agents
§ API
§ Command line tools
§ EC2, S3, SWF, SQS, …