disbatch barcamp buffalo
TRANSCRIPT
![Page 1: Disbatch BarCamp Buffalo](https://reader034.vdocument.in/reader034/viewer/2022051412/5512fccd4a7959df028b4aaa/html5/thumbnails/1.jpg)
DisbatchDistributed batch processing: open-source release
Thursday, June 9, 2011
![Page 2: Disbatch BarCamp Buffalo](https://reader034.vdocument.in/reader034/viewer/2022051412/5512fccd4a7959df028b4aaa/html5/thumbnails/2.jpg)
Distributed Batch Processing
Wanted the best way to manage distributed, long-running tasks
Needed excellent and centralised traceability, logging & visibility
Thursday, June 9, 2011
![Page 3: Disbatch BarCamp Buffalo](https://reader034.vdocument.in/reader034/viewer/2022051412/5512fccd4a7959df028b4aaa/html5/thumbnails/3.jpg)
About us
Matt Busigin - Sr Email Engineer
( a n d o c c a s i o n a l syndicated author on economics)
Matt Berg - Lead Email Engineer, promised to be h e r e , b u t i n s t e a d p r e s e n t l y e n j o y i n g copious vendor-supplied alcohol at MAAWG in SF.
Thursday, June 9, 2011
![Page 4: Disbatch BarCamp Buffalo](https://reader034.vdocument.in/reader034/viewer/2022051412/5512fccd4a7959df028b4aaa/html5/thumbnails/4.jpg)
The ProblemsMass customer migrations
Spamming Sending millions of legitimate pieces of mail
DDOS Load testing
Thursday, June 9, 2011
![Page 5: Disbatch BarCamp Buffalo](https://reader034.vdocument.in/reader034/viewer/2022051412/5512fccd4a7959df028b4aaa/html5/thumbnails/5.jpg)
Requirements
Dependent/chained tasks
Visibility
Distributed & elastic
Platform agnostic
Able to handle large queues
Thursday, June 9, 2011
![Page 6: Disbatch BarCamp Buffalo](https://reader034.vdocument.in/reader034/viewer/2022051412/5512fccd4a7959df028b4aaa/html5/thumbnails/6.jpg)
Features
Durable persistence (using MongoDB as the core)
Multiple active queues
Multiple queue types via plugin architecture
Thursday, June 9, 2011
![Page 7: Disbatch BarCamp Buffalo](https://reader034.vdocument.in/reader034/viewer/2022051412/5512fccd4a7959df028b4aaa/html5/thumbnails/7.jpg)
ArchitectureIt’s all just JSON or BSON!
MongoDB
.NET Disbatch Execution Node
Ruby Disbatch Execution Node
Perl Disbatch Execution Node
Disbatch Command Node
Thursday, June 9, 2011
![Page 8: Disbatch BarCamp Buffalo](https://reader034.vdocument.in/reader034/viewer/2022051412/5512fccd4a7959df028b4aaa/html5/thumbnails/8.jpg)
Per-node ArchitectureConcurrency implementation up to the Disbatch Execution Node implementor
Disbatch Execution Node MongoDB
Queue
Queue
Queue
Thread
Thread
Thread
Thread
Thread
Thursday, June 9, 2011
![Page 9: Disbatch BarCamp Buffalo](https://reader034.vdocument.in/reader034/viewer/2022051412/5512fccd4a7959df028b4aaa/html5/thumbnails/9.jpg)
Disbatch ScreenshotsThe main screen (queues)
Thursday, June 9, 2011
![Page 10: Disbatch BarCamp Buffalo](https://reader034.vdocument.in/reader034/viewer/2022051412/5512fccd4a7959df028b4aaa/html5/thumbnails/10.jpg)
Disbatch ScreenshotsTasks inside of a queue
Thursday, June 9, 2011
![Page 11: Disbatch BarCamp Buffalo](https://reader034.vdocument.in/reader034/viewer/2022051412/5512fccd4a7959df028b4aaa/html5/thumbnails/11.jpg)
Disbatch ScreenshotsTask visibility
Thursday, June 9, 2011
![Page 12: Disbatch BarCamp Buffalo](https://reader034.vdocument.in/reader034/viewer/2022051412/5512fccd4a7959df028b4aaa/html5/thumbnails/12.jpg)
Disbatch ScreenshotsCreating a task from a Mongo Query
Thursday, June 9, 2011
![Page 13: Disbatch BarCamp Buffalo](https://reader034.vdocument.in/reader034/viewer/2022051412/5512fccd4a7959df028b4aaa/html5/thumbnails/13.jpg)
Other cool stuff
We can queue up millions (effectively unlimited) tasks from a Mongo query cursor
Full web interface based on Qooxdoo
Dynamically reload plugin classes (thanks Module::Reload::Selective)
Thursday, June 9, 2011
![Page 14: Disbatch BarCamp Buffalo](https://reader034.vdocument.in/reader034/viewer/2022051412/5512fccd4a7959df028b4aaa/html5/thumbnails/14.jpg)
Other possible uses
Parallel processing of classical maths problems
Fractal generation
Factoring large prime numbers
Any batch/bulk business operation with many different individual task parameters based on a pattern
Thursday, June 9, 2011