Pipes, Arrows, and the UniverseD E C L A R AT I V E T Y P E D S P E C I F I CAT I O N S FO R C O N C U R R E N T A N D C O M P O S A B L E WO R K F L OWS

1

Albert Schimpf

Tools & Toolboxes

Do one thing well

Compose tools together

What about the interface?

2

Universal Interface

3

"Write programs that handle text streams, because that is a universal

interface"

"Expect the output of every program to become the input of

another, as yet unknown, program"

THE BELL SYSTEM TECHNICAL JOURNAL. M. D. MCILROY, ET. AL. “UNIX TIME-SHARING SYSTEM FORWARD”. 1978A QUARTER-CENTURY OF UNIX, PETER H. SALUS, 1994

Pipe = 42

ls -1

wc -l

4

Pipe

Stringp1

Stringp2

5

What's nice?

Separation of business logic and control flow

Composition always works

Good for small chains

Easy (implicit) wiring

6

ShortcomingsConveniently access output previous steps

Graph-structured control flow

Parsing embedded data structures

7

Universal Interface II

"Expect the output of every program to become the input of another, as yet

unknown, program."

Write programs that handle maps1,

because that is a universal interface.

81: HTTPS://WWW.JSON.ORG/JSON-EN.HTML

Workflows COMPOSITION

MAP INTERFACE

9

Workflows COMPOSITION

MAP INTERFACE

10

Map Interface

?

Workflow Systems

Kernmantle ('20), Funflow, Porcupine, iTasks, Luigi, Argo, Oozie, NiFi, Azkaban,

Apache Airflow, AWS Data Pipeline, … (>280) Github

Actions

11HTTPS://GITHUB.COM/COMMON-WORKFLOW-LANGUAGE/COMMON-WORKFLOW-LANGUAGE/WIKI/EXISTING-WORKFLOW-SYSTEMS

WorkflowsCOMPOSITION

MAP INTERFACE

TYPES?

12

Map Interface

1.Chaining tasks 2.Graph of tasks (DAG)

13

Configuration (input, output) of components

Data flow: Map instead of String

p can access any data in the map that accesses p

Transitive implicit dependencies on input possible

ls

wc

p

Configuration

Input Output

14

Configuration (input, output) of components

Data flow: Map instead of String

p can access any data in the map that accesses p

Transitive implicit dependencies on input possible

ls

wc

p

{@x = […]}

{@x = […], @out = 42}

Configuration

Input Output

@x

@x @out

@x@out

15

Graph of Tasks – Semantics

Multiple incoming arrows

Multiple outgoing arrows

Concurrency, isolation??

16

Multiple Outgoing

ArrowsIfThenElse

true

false

λ

λ

17

Isolation:

Dispatched Arrows

Fork

Map

λ

λ

λ

18

Isolation:

Dispatched Arrows

Fork

Map

λ

λ

λ

19

Multiple Incoming

Arrowsλ

20

How to Join?FUTURES!

ForkJoin

?

21

Futures

ForkJoin

join

22

DemoS I M P L E E X A M P L E S & P I P E I N T E R AC T I O N

23

Types!

Atomic values

Lists

Maps

Custom records

Polymorphic types

24

Generated Documentation

25

Design & DemoK E Y WO R D M O N I T O R I N G

26

Summary

The universal string interface is not the only truth

Map interface allows for static type

checking and more

Try it out yourself!

27

Pipes, Arrows, and the UniverseD E C L A R AT I V E T Y P E D S P E C I F I CAT I O N S FO R C O N C U R R E N T A N D C O M P O S A B L E WO R K F L OWS

28

Albert Schimpf

wiki.scraper.server1.link

docs.scraper.server1.link

github.com/scraperflow

hello-world.yml

docker run -v "$PWD":/rt/ --rm albsch/scraper:latest

Install scraper scraper hello-world.yml