Flowex - Flow-Based Programming with Elixir GenStage

Elixir Club 5, Kyiv, January 28, 2017

- Ruby developer at Matic Insurance- Elixir fan- Author and maintainer of

ESpec BDD test framework

github: antonmi

Hello!I am Anton Mishchuk

2

The problem

◎ There are some problems in “conventional programing”:

- code is procedural and sequential- parallelism is not native- hierarchical structure of program- visualisation is more about structure

◎ We need alternative approaches for building our programs to make software better!

3

My previous FBP talk

4

http://www.slideshare.net/AntonMishchuk/flowbased-programming-with-elixir

The goal of the talk

◎ Discuss a special case of Flow-Based Programming: “Railway FBP” (R-FBP)

◎ Present Flowex library which helps to create R-FBP abstractions

◎ Show that Elixir GenStage is not only about data-processing but about software design

5

What will be about

◎ Flow-Based Programming (FBP)◎ Railway Oriented Programming (ROP)◎ Railway FBP◎ Flowex library

Lots of pictures!

6

1.Flow-Based Programming Basic concepts

7

“FBP - a programming paradigm

that defines applications as networks of "black box" processes,

which exchange data across predefined connections by

message passing

J. Paul Morrison, Flow-Based Programming, 2nd Edition 8

FBP diagram

A

B

D

C

IN 1

IN 1

IN 1

IN 2

IN 1

IN 2

OUT 1

OUT 2

OUT 1

OUT 1

OUT 1

9

Express a problem in terms of transforms on streams of data

http://www.jpaulmorrison.com/fbp/FBPnew.ppt

10

Design pros

◎ Independent and reusable components◎ Clean interfaces◎ Simple to reconfigure◎ Minimizes side-effects◎ Designer can sit at one “station”, or can

follow an item through system

http://www.jpaulmorrison.com/fbp/FBPnew.ppt

11

“Native parallelism”

12

2.Railway Oriented Programming Design pattern

13

Express a problem in terms of sequence of functions calls

request |> validate_request|> get_user|> update_db_from_request|> send_email|> return_http_message

14

Define common interface for successful and error cases

◎ For example use: {:ok, "data"} or {:error, "Failure reason"}

◎ Each function returns {:ok, "data"} or {:error, "Failure reason"}

◎ If function is called with {:error, "Failure reason"} it just bypasses it

15

Define interface to bypass errors

http://fsharpforfunandprofit.com/rop/16

So the program is a railway from input to output

http://fsharpforfunandprofit.com/rop/17

Elixir Plug is a good example!

◎ Each “plug” function receives and returns a “connection structure” %Plug.Conn{}

◎ %Plug.Conn{} contains:assigns, cookies, halted, host, method, params, resp_body, resp_cookies, status, and many other attributes

18

3.Railway FBP ROP + FBP

19

Let’s transform each part of “Railway” into FBP component!

20

Place each function into separate process

validate send_emailupdate_db

That is the idea!21

Why Elixir?

22

GenStage

◎ Easiest way to implement a chain of communicating processes

◎ Back-pressure mechanism

producer consumer

Subscribe

Ask

Events

23

Metaprogramming

◎ Simple way for sharing functionality◎ Ability to create expressive DSL

24

4.FlowexRailway FBP

25

Consider an example!

26

Calculate (number + 1) * 2 - 3

27

Note, the functions are cool!

◎ They have the same interface◎ They operate with predefined structWe can join them into pipeline:

28

Let’s add some Flowex magic!

29

… and rename the module to FunPipeline

30

“use Flowex.Pipeline”

◎ Adds ‘pipe’ macro which allows to mark functions “to be placed” into separate Genstage

◎ Defines ‘start’ and ‘stop’ functions for creating and destroying “Flowex pipelines”

◎ Defines ‘run’ function to perform calculations

31

Start FunPipeline!

32

FunPipeline instance

add_one minus_threemult_by_twoproducer consumer

Supervisor

Elixir GenStages

33

%Flowex.Pipeline{} struct

◎ module - the name of the module◎ in_name - unique name of 'producer'◎ out_name - unique name of 'consumer'◎ sup_pid - pid of the pipeline supervisor

34

Run calculations via “run”

Using FunPipeline.run/2 function

Note, ‘a’, ‘b’ and ‘c’ were set!35

Run calculations using Flowex.Client

Using Flowex.Client module

36

How it works in details

add_one

minus_three

mult_by_two

producer

consumer

Flowex.Client

%FunPipeline{number: 2}

%Flowex.IP{struct: %FunPipeline{number: 2}, ...}

number: 2

number: 2

number: 3

number: 6

number: 3number: 3

self()

%FunPipeline{number: 3}

37

What is sync and what is async?

ClientAsynchronouscommunicationself()

Synchronouscommunication

38

Is there no parallelism?

39

There can be a lot of clients!

Client

Client

Client

40

Bottlenecks.What if there is very slow process in the pipeline?

41

Just small changes in the code

42

And you get this

add_one

minus_three

mult_by_twoproducer consumer

mult_by_two

minus_three

mult_by_two

43

Reusable components with “module pipelines”

44

Module must implement just two functions

Like Elixir Plug module45

Pipeline module

46

Conclusion

47

Why it is cool!

48

It is easy to understand

◎ Pipelines explicitly define a structure of data will be processed

◎ Pipelines explicitly define a way the data will come

◎ Pipelines explicitly define a parallel executors structure

49

It is easy to maintain and reuse

◎ There is a predefined set of working processes (components) in a program

◎ Each component is isolated◎ Pipelines can (and should) reuse

components

50

It is about controlled parallelism

◎ One can controll number of clients supplying data to a pipeline

◎ One can control the number of processes available for each component

51

Thank you! Questions?

52

You can star the project here: https://github.com/antonmi/flowex