flow-based programming with elixir
TRANSCRIPT
![Page 1: Flow-based programming with Elixir](https://reader034.vdocument.in/reader034/viewer/2022042502/5882c6a11a28abb2478b6f4d/html5/thumbnails/1.jpg)
Flow-based programming with Elixir
3rd Kyiv Elixir meetup, October 1, 2016
![Page 2: Flow-based programming with Elixir](https://reader034.vdocument.in/reader034/viewer/2022042502/5882c6a11a28abb2478b6f4d/html5/thumbnails/2.jpg)
Hello!
I am Anton Mishchuk
Ruby developer at Matic
Elixir fan
ESpec’s author and maintainer
Github: antonmi
2
![Page 3: Flow-based programming with Elixir](https://reader034.vdocument.in/reader034/viewer/2022042502/5882c6a11a28abb2478b6f4d/html5/thumbnails/3.jpg)
Introduction◈ Elixir new awesome feature - GenStage◈ Elixir Flow-Based Programming (Peter C Marks)◈ Flow-Based Programming, 2nd Edition (J. Paul Morrison)
3
![Page 4: Flow-based programming with Elixir](https://reader034.vdocument.in/reader034/viewer/2022042502/5882c6a11a28abb2478b6f4d/html5/thumbnails/4.jpg)
What I’d tell you
◈ “Conventional programming”◈ Flow-Based Programming◈ “Telegram problem” example◈ Elixir GenStage◈ “Telegram problem” redesign◈ Data-flow routing with GenStage
4
![Page 5: Flow-based programming with Elixir](https://reader034.vdocument.in/reader034/viewer/2022042502/5882c6a11a28abb2478b6f4d/html5/thumbnails/5.jpg)
1.Software design in conventional programming
5
Warning! Too general philosophy!
![Page 6: Flow-based programming with Elixir](https://reader034.vdocument.in/reader034/viewer/2022042502/5882c6a11a28abb2478b6f4d/html5/thumbnails/6.jpg)
Micro level
6
◈ Basic components:○ Objects in OOP comprising both
information and behaviour○ Functions and data-structures in FP
◈ Hierarchical organization◈ Sequential communication or
evaluation
![Page 7: Flow-based programming with Elixir](https://reader034.vdocument.in/reader034/viewer/2022042502/5882c6a11a28abb2478b6f4d/html5/thumbnails/7.jpg)
DSLs and Frameworksas “coordination languages”
◈ Define common abstractions◈ Initialize your components◈ Use components in right order
7
![Page 8: Flow-based programming with Elixir](https://reader034.vdocument.in/reader034/viewer/2022042502/5882c6a11a28abb2478b6f4d/html5/thumbnails/8.jpg)
Visualization
8
![Page 9: Flow-based programming with Elixir](https://reader034.vdocument.in/reader034/viewer/2022042502/5882c6a11a28abb2478b6f4d/html5/thumbnails/9.jpg)
Concurrency& parallelism
9
![Page 10: Flow-based programming with Elixir](https://reader034.vdocument.in/reader034/viewer/2022042502/5882c6a11a28abb2478b6f4d/html5/thumbnails/10.jpg)
◈ Hierarchic structure of program◈ Code is procedural and sequential◈ Visualisation is more about structure◈ Concurrency is not native
What is wrong with conventional programming?
10
![Page 11: Flow-based programming with Elixir](https://reader034.vdocument.in/reader034/viewer/2022042502/5882c6a11a28abb2478b6f4d/html5/thumbnails/11.jpg)
Basic concepts
2.Flow-Based Programming
11
![Page 12: Flow-based programming with Elixir](https://reader034.vdocument.in/reader034/viewer/2022042502/5882c6a11a28abb2478b6f4d/html5/thumbnails/12.jpg)
“FBP - a programming paradigm that defines applications as networks of "black box" processes,
which exchange data across predefined connections by message passing, where the connections are specified externally to the processes. These black box processes
can be reconnected endlessly to form different applications without having to be changed internally.
https://en.wikipedia.org/wiki/Flow-based_programming
12
![Page 13: Flow-based programming with Elixir](https://reader034.vdocument.in/reader034/viewer/2022042502/5882c6a11a28abb2478b6f4d/html5/thumbnails/13.jpg)
FBP diagram
A
B
D
C
IN 1
IN 1
IN 1
IN 2
IN 1
IN 2
OUT 1
OUT 2
OUT 1
OUT 1
OUT 1
Processes, ports, connections
13
![Page 14: Flow-based programming with Elixir](https://reader034.vdocument.in/reader034/viewer/2022042502/5882c6a11a28abb2478b6f4d/html5/thumbnails/14.jpg)
“... whereas the conventional approaches to programming start with process and view data as
secondary, business applications are usually designed starting with data and viewing process as
secondary – processes are just the way data is created, manipulated and destroyed.
J. Paul Morrison, Flow-Based Programming, 2nd Edition
14
![Page 15: Flow-based programming with Elixir](https://reader034.vdocument.in/reader034/viewer/2022042502/5882c6a11a28abb2478b6f4d/html5/thumbnails/15.jpg)
Express a problem in terms of transforms on
streams of data15
![Page 16: Flow-based programming with Elixir](https://reader034.vdocument.in/reader034/viewer/2022042502/5882c6a11a28abb2478b6f4d/html5/thumbnails/16.jpg)
Soft Drink Bottling Factory
◈ Independent well-defined components◈ Clean interfaces◈ Simple to reconfigure◈ Minimizes side-effects◈ Designer can sit at one “station”, or can
follow an item through system
http://www.jpaulmorrison.com/fbp/FBPnew.ppt16
![Page 17: Flow-based programming with Elixir](https://reader034.vdocument.in/reader034/viewer/2022042502/5882c6a11a28abb2478b6f4d/html5/thumbnails/17.jpg)
FPB characteristics
◈ Asynchronous processes communicating via streams of data packets
◈ Data packets with a lifetime of their own◈ Definition of connections external to
components◈ Consistent view from macro to micro
http://www.jpaulmorrison.com/fbp/FBPnew.ppt17
![Page 18: Flow-based programming with Elixir](https://reader034.vdocument.in/reader034/viewer/2022042502/5882c6a11a28abb2478b6f4d/html5/thumbnails/18.jpg)
Native parallelism
18
![Page 19: Flow-based programming with Elixir](https://reader034.vdocument.in/reader034/viewer/2022042502/5882c6a11a28abb2478b6f4d/html5/thumbnails/19.jpg)
FBP is about “coordination
language”19
![Page 20: Flow-based programming with Elixir](https://reader034.vdocument.in/reader034/viewer/2022042502/5882c6a11a28abb2478b6f4d/html5/thumbnails/20.jpg)
Everything new is actually well-forgotten old
◈ 1971, IBM, basic concepts◈ 1975, Bank of Montreal, on-line banking
system, still works!◈ 1994, Flow-Based Programming, 1st Ed◈ 2010, Flow-Based Programming, 2st Ed◈ 2012, NoFlo project by Henri Bergius,FBP in JavaScript◈ JavaFBP, C#FBP, CppFBP (C++ and Lua)
20
![Page 21: Flow-based programming with Elixir](https://reader034.vdocument.in/reader034/viewer/2022042502/5882c6a11a28abb2478b6f4d/html5/thumbnails/21.jpg)
What about Erlang/Elixir?
21
![Page 22: Flow-based programming with Elixir](https://reader034.vdocument.in/reader034/viewer/2022042502/5882c6a11a28abb2478b6f4d/html5/thumbnails/22.jpg)
FBP example with Elixir
3.Telegram problem
22
![Page 23: Flow-based programming with Elixir](https://reader034.vdocument.in/reader034/viewer/2022042502/5882c6a11a28abb2478b6f4d/html5/thumbnails/23.jpg)
Telegram problem
A program which accepts lines of text and generates output lines of a different predefined length, without splitting any of the words in the text.The program accepts an input stream of lines of some length and produce an output stream of lines of another length.
23
![Page 24: Flow-based programming with Elixir](https://reader034.vdocument.in/reader034/viewer/2022042502/5882c6a11a28abb2478b6f4d/html5/thumbnails/24.jpg)
Problem FBP design using GenServers
ReadSeq Recompose WriteSeqDecompose
Lines Words Lines
◈ Each component is a GenServer. ◈ Each component implements only
one function - “run” which transforms input to output
◈ Communication is asynchronous◈ Code is here:
https://github.com/antonmi/kyiv_meetup_324
![Page 25: Flow-based programming with Elixir](https://reader034.vdocument.in/reader034/viewer/2022042502/5882c6a11a28abb2478b6f4d/html5/thumbnails/25.jpg)
ReadSeq (naive implementation)
25
![Page 26: Flow-based programming with Elixir](https://reader034.vdocument.in/reader034/viewer/2022042502/5882c6a11a28abb2478b6f4d/html5/thumbnails/26.jpg)
Decompose (naive implementation)
26
![Page 27: Flow-based programming with Elixir](https://reader034.vdocument.in/reader034/viewer/2022042502/5882c6a11a28abb2478b6f4d/html5/thumbnails/27.jpg)
Recompose (naive implementation)
27
![Page 28: Flow-based programming with Elixir](https://reader034.vdocument.in/reader034/viewer/2022042502/5882c6a11a28abb2478b6f4d/html5/thumbnails/28.jpg)
WriteSeq (naive implementation)
28
![Page 29: Flow-based programming with Elixir](https://reader034.vdocument.in/reader034/viewer/2022042502/5882c6a11a28abb2478b6f4d/html5/thumbnails/29.jpg)
The main Telegram module(naive implementation)
29
![Page 30: Flow-based programming with Elixir](https://reader034.vdocument.in/reader034/viewer/2022042502/5882c6a11a28abb2478b6f4d/html5/thumbnails/30.jpg)
What is cool in this simple implementation
◈ Each component do its own part of work◈ The only API is “init” and “run”◈ Components works in parallel◈ Components are under supervisor
30
![Page 31: Flow-based programming with Elixir](https://reader034.vdocument.in/reader034/viewer/2022042502/5882c6a11a28abb2478b6f4d/html5/thumbnails/31.jpg)
What is wrong with the implementation
◈ Components are not independent (each of them knows who is the next)
◈ Flow is controlled internally by each of the component
◈ There is no back-pressure mechanism (when some component lags it can be overflowed by data)
31
![Page 32: Flow-based programming with Elixir](https://reader034.vdocument.in/reader034/viewer/2022042502/5882c6a11a28abb2478b6f4d/html5/thumbnails/32.jpg)
Experimental feature.
4.Elixir GenStage
32
![Page 33: Flow-based programming with Elixir](https://reader034.vdocument.in/reader034/viewer/2022042502/5882c6a11a28abb2478b6f4d/html5/thumbnails/33.jpg)
GenStage abstractions
◈ GenStage is build on top of GenServer◈ Each of the stage can be producer,
consumer or both
producer producer consumer consumerproducer
consumer
33
![Page 34: Flow-based programming with Elixir](https://reader034.vdocument.in/reader034/viewer/2022042502/5882c6a11a28abb2478b6f4d/html5/thumbnails/34.jpg)
GenStage communication
◈ Consumer subscribes to producer◈ Consumer asks for data (sends a demand)◈ Producer sends data
producer consumer
Subscribe
Ask
Events
34
![Page 35: Flow-based programming with Elixir](https://reader034.vdocument.in/reader034/viewer/2022042502/5882c6a11a28abb2478b6f4d/html5/thumbnails/35.jpg)
GenStage API
◈ init(state) which must return:○ {:producer, state}○ {:producer_consumer, state}○ {:consumer, state}
◈ handle_demand(events, from, state)○ {:noreply, events, new_state}
◈ handle_events(events, from, state)○ {:noreply, events, new_state}
35
![Page 36: Flow-based programming with Elixir](https://reader034.vdocument.in/reader034/viewer/2022042502/5882c6a11a28abb2478b6f4d/html5/thumbnails/36.jpg)
5.Telegram problem redesign
36
![Page 37: Flow-based programming with Elixir](https://reader034.vdocument.in/reader034/viewer/2022042502/5882c6a11a28abb2478b6f4d/html5/thumbnails/37.jpg)
ReadSeq GenStage
37
![Page 38: Flow-based programming with Elixir](https://reader034.vdocument.in/reader034/viewer/2022042502/5882c6a11a28abb2478b6f4d/html5/thumbnails/38.jpg)
Decompose GenStage
38
![Page 39: Flow-based programming with Elixir](https://reader034.vdocument.in/reader034/viewer/2022042502/5882c6a11a28abb2478b6f4d/html5/thumbnails/39.jpg)
Recompose GenStage
39
![Page 40: Flow-based programming with Elixir](https://reader034.vdocument.in/reader034/viewer/2022042502/5882c6a11a28abb2478b6f4d/html5/thumbnails/40.jpg)
WriteSeq GenStage
40
![Page 41: Flow-based programming with Elixir](https://reader034.vdocument.in/reader034/viewer/2022042502/5882c6a11a28abb2478b6f4d/html5/thumbnails/41.jpg)
Main Telegram module
41
![Page 42: Flow-based programming with Elixir](https://reader034.vdocument.in/reader034/viewer/2022042502/5882c6a11a28abb2478b6f4d/html5/thumbnails/42.jpg)
It is awesome!
◈ Each component is independent!◈ Data flow is coordinated externally!◈ There is a back-pressure!
42
![Page 43: Flow-based programming with Elixir](https://reader034.vdocument.in/reader034/viewer/2022042502/5882c6a11a28abb2478b6f4d/html5/thumbnails/43.jpg)
5.Data-flow routingwith GenStage
43
![Page 44: Flow-based programming with Elixir](https://reader034.vdocument.in/reader034/viewer/2022042502/5882c6a11a28abb2478b6f4d/html5/thumbnails/44.jpg)
GenStage Dispatchers
◈ GenStage.DemandDispatcher○ sends events to the highest demand
◈ GenStage.BroadcastDispatcher○ accumulates demand from all
consumers before broadcasting events to all of them
◈ GenStage.PartitionDispatcher○ sends events according to partitions
44
![Page 45: Flow-based programming with Elixir](https://reader034.vdocument.in/reader034/viewer/2022042502/5882c6a11a28abb2478b6f4d/html5/thumbnails/45.jpg)
Duplicate file example
45
ReadSeq
WriteSeq
Broadcast
WriteSeq
Code is here: https://github.com/antonmi/kyiv_meetup_3
![Page 46: Flow-based programming with Elixir](https://reader034.vdocument.in/reader034/viewer/2022042502/5882c6a11a28abb2478b6f4d/html5/thumbnails/46.jpg)
The only new component is Broadcast
46
![Page 47: Flow-based programming with Elixir](https://reader034.vdocument.in/reader034/viewer/2022042502/5882c6a11a28abb2478b6f4d/html5/thumbnails/47.jpg)
Split file example
47
Code is here: https://github.com/antonmi/kyiv_meetup_3
ReadSeq
WriteSeq
Split
WriteSeq
![Page 48: Flow-based programming with Elixir](https://reader034.vdocument.in/reader034/viewer/2022042502/5882c6a11a28abb2478b6f4d/html5/thumbnails/48.jpg)
The only new component is Split
48
![Page 49: Flow-based programming with Elixir](https://reader034.vdocument.in/reader034/viewer/2022042502/5882c6a11a28abb2478b6f4d/html5/thumbnails/49.jpg)
6.Conclusion
49
![Page 50: Flow-based programming with Elixir](https://reader034.vdocument.in/reader034/viewer/2022042502/5882c6a11a28abb2478b6f4d/html5/thumbnails/50.jpg)
Erlang VM+
Elixir GenStage=
FBP paradigm50
![Page 51: Flow-based programming with Elixir](https://reader034.vdocument.in/reader034/viewer/2022042502/5882c6a11a28abb2478b6f4d/html5/thumbnails/51.jpg)
Thanks!