Download - Ricon/West 2013: Adventures with Riak Pipe
![Page 1: Ricon/West 2013: Adventures with Riak Pipe](https://reader034.vdocument.in/reader034/viewer/2022042814/5550f698b4c90572478b466e/html5/thumbnails/1.jpg)
DISTRIBUTING WORK ACROSSCLUSTERSAdventures With Riak Pipe
Last Updated: October 31, 2013
Susan Potter @SusanPotter
![Page 2: Ricon/West 2013: Adventures with Riak Pipe](https://reader034.vdocument.in/reader034/viewer/2022042814/5550f698b4c90572478b466e/html5/thumbnails/2.jpg)
Background Concepts & Intuitions Applications
WHOAMI
2
![Page 3: Ricon/West 2013: Adventures with Riak Pipe](https://reader034.vdocument.in/reader034/viewer/2022042814/5550f698b4c90572478b466e/html5/thumbnails/3.jpg)
..
WHOAMI..
2013-10-31
Distributing Work Across Clusters
whoami
An app + middleware dev who has felt Ops pain too
![Page 4: Ricon/West 2013: Adventures with Riak Pipe](https://reader034.vdocument.in/reader034/viewer/2022042814/5550f698b4c90572478b466e/html5/thumbnails/4.jpg)
BACKGROUND
![Page 5: Ricon/West 2013: Adventures with Riak Pipe](https://reader034.vdocument.in/reader034/viewer/2022042814/5550f698b4c90572478b466e/html5/thumbnails/5.jpg)
Background Concepts & Intuitions Applications
MODEL/ASSUMPTIONS
4
![Page 6: Ricon/West 2013: Adventures with Riak Pipe](https://reader034.vdocument.in/reader034/viewer/2022042814/5550f698b4c90572478b466e/html5/thumbnails/6.jpg)
Background Concepts & Intuitions Applications
MOTIVATING USES
5
![Page 7: Ricon/West 2013: Adventures with Riak Pipe](https://reader034.vdocument.in/reader034/viewer/2022042814/5550f698b4c90572478b466e/html5/thumbnails/7.jpg)
Background Concepts & Intuitions Applications
COMMON APPROACHES
6
![Page 8: Ricon/West 2013: Adventures with Riak Pipe](https://reader034.vdocument.in/reader034/viewer/2022042814/5550f698b4c90572478b466e/html5/thumbnails/8.jpg)
Background Concepts & Intuitions Applications
WHY DISTRIBUTE? WHY DECENTRALIZE?
→ Probability of system failure→ Large datasets (data locality)→ Eliminate SPOFs→ Commodity hardware
7
![Page 9: Ricon/West 2013: Adventures with Riak Pipe](https://reader034.vdocument.in/reader034/viewer/2022042814/5550f698b4c90572478b466e/html5/thumbnails/9.jpg)
Background Concepts & Intuitions Applications
8
![Page 10: Ricon/West 2013: Adventures with Riak Pipe](https://reader034.vdocument.in/reader034/viewer/2022042814/5550f698b4c90572478b466e/html5/thumbnails/10.jpg)
CONCEPTS & INTUITIONS
![Page 11: Ricon/West 2013: Adventures with Riak Pipe](https://reader034.vdocument.in/reader034/viewer/2022042814/5550f698b4c90572478b466e/html5/thumbnails/11.jpg)
Background Concepts & Intuitions Applications
LAYERS
10
![Page 12: Ricon/West 2013: Adventures with Riak Pipe](https://reader034.vdocument.in/reader034/viewer/2022042814/5550f698b4c90572478b466e/html5/thumbnails/12.jpg)
Background Concepts & Intuitions Applications
TERMINOLOGY (CORE)
→ Partition→ Vnode→ Hinted Handoff→ Read Repair
11
![Page 13: Ricon/West 2013: Adventures with Riak Pipe](https://reader034.vdocument.in/reader034/viewer/2022042814/5550f698b4c90572478b466e/html5/thumbnails/13.jpg)
Background Concepts & Intuitions Applications
TERMINOLOGY (PIPE)
→ Pipe→ Fitting ("phase")→ Queue→ Worker
12
![Page 14: Ricon/West 2013: Adventures with Riak Pipe](https://reader034.vdocument.in/reader034/viewer/2022042814/5550f698b4c90572478b466e/html5/thumbnails/14.jpg)
Background Concepts & Intuitions Applications
EXAMPLE UNIX PIPE
1 find . -name "*.rb" \2 | xargs egrep "#.*?TODO:" \3 | wc -l
Character-based, through file descriptors
13
![Page 15: Ricon/West 2013: Adventures with Riak Pipe](https://reader034.vdocument.in/reader034/viewer/2022042814/5550f698b4c90572478b466e/html5/thumbnails/15.jpg)
Background Concepts & Intuitions Applications
EXAMPLE FUNCTION COMPOSITION
1 (length . mapToUpper . sanitize) input
Value based, through functions
14
![Page 16: Ricon/West 2013: Adventures with Riak Pipe](https://reader034.vdocument.in/reader034/viewer/2022042814/5550f698b4c90572478b466e/html5/thumbnails/16.jpg)
Background Concepts & Intuitions Applications
EXAMPLE RIAK PIPE
1 [ #fitting_spec{ name=fetch_trades2 , module=riskmgr_fetch_trades3 , ...}4 , #fitting_spec{ name=calc_var5 , module=riskmgr_calc_var6 , ...}7 ]
Message-based, across nodes
15
![Page 17: Ricon/West 2013: Adventures with Riak Pipe](https://reader034.vdocument.in/reader034/viewer/2022042814/5550f698b4c90572478b466e/html5/thumbnails/17.jpg)
Background Concepts & Intuitions Applications
VNODE AND WORKERS
→ vnodemanages lifecycle and queues of workers
→ vnode workerprocesses inputs
16
![Page 18: Ricon/West 2013: Adventures with Riak Pipe](https://reader034.vdocument.in/reader034/viewer/2022042814/5550f698b4c90572478b466e/html5/thumbnails/18.jpg)
Background Concepts & Intuitions Applications
VNODE WORKER BEHAVIOR
1 behaviour_info(callbacks) ->2 [{init,2},3 {process ,3},4 {done,1}];5 behaviour_info(_Other) ->6 undefined.7 %% Optionally two more too, not required.
17
![Page 19: Ricon/West 2013: Adventures with Riak Pipe](https://reader034.vdocument.in/reader034/viewer/2022042814/5550f698b4c90572478b466e/html5/thumbnails/19.jpg)
Background Concepts & Intuitions Applications
DISTRIBUTION: CHASHFUN
→ Use random uniform hash fun to saturate workers→ See Bryan Fink's previous Pipe presentations
18
![Page 20: Ricon/West 2013: Adventures with Riak Pipe](https://reader034.vdocument.in/reader034/viewer/2022042814/5550f698b4c90572478b466e/html5/thumbnails/20.jpg)
Background Concepts & Intuitions Applications
VALIDATION
1 % #fitting_spec { module = ..., arg = 43, ...}
2 % validate_arg/13 validate_arg(Arg) when is_integer(Arg) -> ok;4 validate_arg(Arg) when is_atom(Arg) -> ok;5 validate_arg(_) -> {error, "Argument must be
a valid integer or atom"}.
19
![Page 21: Ricon/West 2013: Adventures with Riak Pipe](https://reader034.vdocument.in/reader034/viewer/2022042814/5550f698b4c90572478b466e/html5/thumbnails/21.jpg)
Background Concepts & Intuitions Applications
LEAKY PIPES
http://www.flickr.com/photos/thirteenofclubs/
20
![Page 22: Ricon/West 2013: Adventures with Riak Pipe](https://reader034.vdocument.in/reader034/viewer/2022042814/5550f698b4c90572478b466e/html5/thumbnails/22.jpg)
Background Concepts & Intuitions Applications
RIAK (CORE) MECHANICS
→ HandoffMigrate vnodes from node to node
1 % Old Node2 % Called with last known State of worker3 archive/1 % returns {ok, Archive}4
5 % New Node6 % Called with Archive and State from old node7 handoff/2 % returns {ok, NewState}
21
![Page 23: Ricon/West 2013: Adventures with Riak Pipe](https://reader034.vdocument.in/reader034/viewer/2022042814/5550f698b4c90572478b466e/html5/thumbnails/23.jpg)
Background Concepts & Intuitions Applications
FAILURES
→ validate_argSeen above
→ Pipe client process diesYOLO
→ nvalHow many vnodes to ask before failing, default 1
→ vnode worker logicThink using dataflow semantics
22
![Page 24: Ricon/West 2013: Adventures with Riak Pipe](https://reader034.vdocument.in/reader034/viewer/2022042814/5550f698b4c90572478b466e/html5/thumbnails/24.jpg)
Background Concepts & Intuitions Applications
EXISTING FITTINGS
→ riak_pipe_w_teeUseful for intermediate results
→ riak_pipe_w_xformSimple delegater: 3-arity function
→ riak_pipe_w_reduceWhat you would expect: simple accumulating reduce
→ riak_kv_pipe_getTake advantage of data locality with cohosted KV store
23
![Page 25: Ricon/West 2013: Adventures with Riak Pipe](https://reader034.vdocument.in/reader034/viewer/2022042814/5550f698b4c90572478b466e/html5/thumbnails/25.jpg)
APPLICATIONS
![Page 26: Ricon/West 2013: Adventures with Riak Pipe](https://reader034.vdocument.in/reader034/viewer/2022042814/5550f698b4c90572478b466e/html5/thumbnails/26.jpg)
Background Concepts & Intuitions Applications
KNOWN USES
→ Riak's Map/Reduce→ Risk metrics→ Tenant Usage
25
![Page 27: Ricon/West 2013: Adventures with Riak Pipe](https://reader034.vdocument.in/reader034/viewer/2022042814/5550f698b4c90572478b466e/html5/thumbnails/27.jpg)
Background Concepts & Intuitions Applications
TROUBLESHOOTING
→ riak_pipe:status/1Provides fittings, processed, failures, queue_length, etc stats
→ riak_pipe_w_crash fittingUsed to test Riak Pipe
→ riak_pipe:active_pipelines/1See all active pipelines
→ riak_pipe_cinfo modulecluster info interrogation module
26
![Page 28: Ricon/West 2013: Adventures with Riak Pipe](https://reader034.vdocument.in/reader034/viewer/2022042814/5550f698b4c90572478b466e/html5/thumbnails/28.jpg)
Background Concepts & Intuitions Applications
FURTHER WORK
→ More applications (e.g. genetics, 3rd party APIs with ratelimits)
→ Decentralized pipe control→ Measure "completeness" of resultset
27
![Page 29: Ricon/West 2013: Adventures with Riak Pipe](https://reader034.vdocument.in/reader034/viewer/2022042814/5550f698b4c90572478b466e/html5/thumbnails/29.jpg)
Background Concepts & Intuitions Applications
RELATED WORK
→ "Pipe" libraries EVERYWHERE (pipe, scalaz-stream)→ riak_pg→ Map-Reduce frameworks→ Staged Event Driven Architecture→ Event Stream Processing→ Dataflow multi-stage processing
28
![Page 30: Ricon/West 2013: Adventures with Riak Pipe](https://reader034.vdocument.in/reader034/viewer/2022042814/5550f698b4c90572478b466e/html5/thumbnails/30.jpg)
Background Concepts & Intuitions Applications
ROYAL FAIL
http://www.flickr.com/photos/dadavidov/
29
![Page 31: Ricon/West 2013: Adventures with Riak Pipe](https://reader034.vdocument.in/reader034/viewer/2022042814/5550f698b4c90572478b466e/html5/thumbnails/31.jpg)
Background Concepts & Intuitions Applications
QUESTIONS
Questions?
30