Flowex - Flow-Based Programming with Elixir GenStage
Elixir Club 5, Kyiv, January 28, 2017
- Ruby developer at Matic Insurance- Elixir fan- Author and maintainer of
ESpec BDD test framework
github: antonmi
Hello!I am Anton Mishchuk
2
The problem
◎ There are some problems in “conventional programing”:
- code is procedural and sequential- parallelism is not native- hierarchical structure of program- visualisation is more about structure
◎ We need alternative approaches for building our programs to make software better!
3
My previous FBP talk
4
http://www.slideshare.net/AntonMishchuk/flowbased-programming-with-elixir
The goal of the talk
◎ Discuss a special case of Flow-Based Programming: “Railway FBP” (R-FBP)
◎ Present Flowex library which helps to create R-FBP abstractions
◎ Show that Elixir GenStage is not only about data-processing but about software design
5
What will be about
◎ Flow-Based Programming (FBP)◎ Railway Oriented Programming (ROP)◎ Railway FBP◎ Flowex library
Lots of pictures!
6
1.Flow-Based Programming Basic concepts
7
“FBP - a programming paradigm
that defines applications as networks of "black box" processes,
which exchange data across predefined connections by
message passing
J. Paul Morrison, Flow-Based Programming, 2nd Edition 8
FBP diagram
A
B
D
C
IN 1
IN 1
IN 1
IN 2
IN 1
IN 2
OUT 1
OUT 2
OUT 1
OUT 1
OUT 1
9
Express a problem in terms of transforms on streams of data
http://www.jpaulmorrison.com/fbp/FBPnew.ppt
10
Design pros
◎ Independent and reusable components◎ Clean interfaces◎ Simple to reconfigure◎ Minimizes side-effects◎ Designer can sit at one “station”, or can
follow an item through system
http://www.jpaulmorrison.com/fbp/FBPnew.ppt
11
“Native parallelism”
12
2.Railway Oriented Programming Design pattern
13
Express a problem in terms of sequence of functions calls
request |> validate_request|> get_user|> update_db_from_request|> send_email|> return_http_message
14
Define common interface for successful and error cases
◎ For example use: {:ok, "data"} or {:error, "Failure reason"}
◎ Each function returns {:ok, "data"} or {:error, "Failure reason"}
◎ If function is called with {:error, "Failure reason"} it just bypasses it
15
Define interface to bypass errors
http://fsharpforfunandprofit.com/rop/16
So the program is a railway from input to output
http://fsharpforfunandprofit.com/rop/17
Elixir Plug is a good example!
◎ Each “plug” function receives and returns a “connection structure” %Plug.Conn{}
◎ %Plug.Conn{} contains:assigns, cookies, halted, host, method, params, resp_body, resp_cookies, status, and many other attributes
18
3.Railway FBP ROP + FBP
19
Let’s transform each part of “Railway” into FBP component!
20
Place each function into separate process
validate send_emailupdate_db
That is the idea!21
Why Elixir?
22
GenStage
◎ Easiest way to implement a chain of communicating processes
◎ Back-pressure mechanism
producer consumer
Subscribe
Ask
Events
23
Metaprogramming
◎ Simple way for sharing functionality◎ Ability to create expressive DSL
24
4.FlowexRailway FBP
25
Consider an example!
26
Calculate (number + 1) * 2 - 3
27
Note, the functions are cool!
◎ They have the same interface◎ They operate with predefined structWe can join them into pipeline:
28
Let’s add some Flowex magic!
29
… and rename the module to FunPipeline
30
“use Flowex.Pipeline”
◎ Adds ‘pipe’ macro which allows to mark functions “to be placed” into separate Genstage
◎ Defines ‘start’ and ‘stop’ functions for creating and destroying “Flowex pipelines”
◎ Defines ‘run’ function to perform calculations
31
Start FunPipeline!
32
FunPipeline instance
add_one minus_threemult_by_twoproducer consumer
Supervisor
Elixir GenStages
33
%Flowex.Pipeline{} struct
◎ module - the name of the module◎ in_name - unique name of 'producer'◎ out_name - unique name of 'consumer'◎ sup_pid - pid of the pipeline supervisor
34
Run calculations via “run”
Using FunPipeline.run/2 function
Note, ‘a’, ‘b’ and ‘c’ were set!35
Run calculations using Flowex.Client
Using Flowex.Client module
36
How it works in details
add_one
minus_three
mult_by_two
producer
consumer
Flowex.Client
%FunPipeline{number: 2}
%Flowex.IP{struct: %FunPipeline{number: 2}, ...}
number: 2
number: 2
number: 3
number: 6
number: 3number: 3
self()
%FunPipeline{number: 3}
37
What is sync and what is async?
ClientAsynchronouscommunicationself()
Synchronouscommunication
38
Is there no parallelism?
39
There can be a lot of clients!
Client
Client
Client
40
Bottlenecks.What if there is very slow process in the pipeline?
41
Just small changes in the code
42
And you get this
add_one
minus_three
mult_by_twoproducer consumer
mult_by_two
minus_three
mult_by_two
43
Reusable components with “module pipelines”
44
Module must implement just two functions
Like Elixir Plug module45
Pipeline module
46
Conclusion
47
Why it is cool!
48
It is easy to understand
◎ Pipelines explicitly define a structure of data will be processed
◎ Pipelines explicitly define a way the data will come
◎ Pipelines explicitly define a parallel executors structure
49
It is easy to maintain and reuse
◎ There is a predefined set of working processes (components) in a program
◎ Each component is isolated◎ Pipelines can (and should) reuse
components
50
It is about controlled parallelism
◎ One can controll number of clients supplying data to a pipeline
◎ One can control the number of processes available for each component
51
Thank you! Questions?
52
You can star the project here: https://github.com/antonmi/flowex