Download - The Architecture of Wemlin Hub
![Page 1: The Architecture of Wemlin Hub](https://reader036.vdocument.in/reader036/viewer/2022070321/558e46c91a28ab51268b46c2/html5/thumbnails/1.jpg)
The Architecture of Wemlin HubOgnen Ivanovski, Netcetera Jazoon ‘14
![Page 2: The Architecture of Wemlin Hub](https://reader036.vdocument.in/reader036/viewer/2022070321/558e46c91a28ab51268b46c2/html5/thumbnails/2.jpg)
Wemlin
![Page 3: The Architecture of Wemlin Hub](https://reader036.vdocument.in/reader036/viewer/2022070321/558e46c91a28ab51268b46c2/html5/thumbnails/3.jpg)
![Page 4: The Architecture of Wemlin Hub](https://reader036.vdocument.in/reader036/viewer/2022070321/558e46c91a28ab51268b46c2/html5/thumbnails/4.jpg)
![Page 5: The Architecture of Wemlin Hub](https://reader036.vdocument.in/reader036/viewer/2022070321/558e46c91a28ab51268b46c2/html5/thumbnails/5.jpg)
Data
![Page 6: The Architecture of Wemlin Hub](https://reader036.vdocument.in/reader036/viewer/2022070321/558e46c91a28ab51268b46c2/html5/thumbnails/6.jpg)
Data
Planning Software
![Page 7: The Architecture of Wemlin Hub](https://reader036.vdocument.in/reader036/viewer/2022070321/558e46c91a28ab51268b46c2/html5/thumbnails/7.jpg)
Planning Software
![Page 8: The Architecture of Wemlin Hub](https://reader036.vdocument.in/reader036/viewer/2022070321/558e46c91a28ab51268b46c2/html5/thumbnails/8.jpg)
AVCS
![Page 9: The Architecture of Wemlin Hub](https://reader036.vdocument.in/reader036/viewer/2022070321/558e46c91a28ab51268b46c2/html5/thumbnails/9.jpg)
AVCS
![Page 10: The Architecture of Wemlin Hub](https://reader036.vdocument.in/reader036/viewer/2022070321/558e46c91a28ab51268b46c2/html5/thumbnails/10.jpg)
Wemlin Hub
![Page 11: The Architecture of Wemlin Hub](https://reader036.vdocument.in/reader036/viewer/2022070321/558e46c91a28ab51268b46c2/html5/thumbnails/11.jpg)
gWemlin Hub
![Page 12: The Architecture of Wemlin Hub](https://reader036.vdocument.in/reader036/viewer/2022070321/558e46c91a28ab51268b46c2/html5/thumbnails/12.jpg)
Plan Data
transport schedule (plan) over certain time
package in nature
Formats: HAFAS, GTFS, VDV 453 REF, VDV 454 REF, REST
![Page 13: The Architecture of Wemlin Hub](https://reader036.vdocument.in/reader036/viewer/2022070321/558e46c91a28ab51268b46c2/html5/thumbnails/13.jpg)
RT Streams
updates on planned data
stream in nature
Formats: GTFS RT, VDV-453, VDV-454, REST
![Page 14: The Architecture of Wemlin Hub](https://reader036.vdocument.in/reader036/viewer/2022070321/558e46c91a28ab51268b46c2/html5/thumbnails/14.jpg)
What does the hub do
Gets the data from the source
Figures out what the hell the source is talking about
Stores the data and passes it to the clients
![Page 15: The Architecture of Wemlin Hub](https://reader036.vdocument.in/reader036/viewer/2022070321/558e46c91a28ab51268b46c2/html5/thumbnails/15.jpg)
![Page 16: The Architecture of Wemlin Hub](https://reader036.vdocument.in/reader036/viewer/2022070321/558e46c91a28ab51268b46c2/html5/thumbnails/16.jpg)
Getting Data
different protocols & implementations
remote system availability issues
![Page 17: The Architecture of Wemlin Hub](https://reader036.vdocument.in/reader036/viewer/2022070321/558e46c91a28ab51268b46c2/html5/thumbnails/17.jpg)
Figuring things out
different sources, no referential guarantees
every data source is different
data quality issues
Example: station refeferences
Example: trip references
![Page 18: The Architecture of Wemlin Hub](https://reader036.vdocument.in/reader036/viewer/2022070321/558e46c91a28ab51268b46c2/html5/thumbnails/18.jpg)
Data Enrichment
line colors
location
station metadata (e.g. which lines stop there, which stations are near by)
![Page 19: The Architecture of Wemlin Hub](https://reader036.vdocument.in/reader036/viewer/2022070321/558e46c91a28ab51268b46c2/html5/thumbnails/19.jpg)
Serve many users
![Page 20: The Architecture of Wemlin Hub](https://reader036.vdocument.in/reader036/viewer/2022070321/558e46c91a28ab51268b46c2/html5/thumbnails/20.jpg)
Summary
standardized data formats (but implementations vary)
disparate systems (referencing is hard)
enrichment
![Page 21: The Architecture of Wemlin Hub](https://reader036.vdocument.in/reader036/viewer/2022070321/558e46c91a28ab51268b46c2/html5/thumbnails/21.jpg)
Key ConcernsRT
low latency
throughput
available
excllent failure management !
Batch
throughput
![Page 22: The Architecture of Wemlin Hub](https://reader036.vdocument.in/reader036/viewer/2022070321/558e46c91a28ab51268b46c2/html5/thumbnails/22.jpg)
Key Concerns
processing programs the same for both batch and RT data
![Page 23: The Architecture of Wemlin Hub](https://reader036.vdocument.in/reader036/viewer/2022070321/558e46c91a28ab51268b46c2/html5/thumbnails/23.jpg)
Key Concerns
fast reaction to load changes
![Page 24: The Architecture of Wemlin Hub](https://reader036.vdocument.in/reader036/viewer/2022070321/558e46c91a28ab51268b46c2/html5/thumbnails/24.jpg)
Architecture
Constraints
decisions hard to change
![Page 25: The Architecture of Wemlin Hub](https://reader036.vdocument.in/reader036/viewer/2022070321/558e46c91a28ab51268b46c2/html5/thumbnails/25.jpg)
![Page 26: The Architecture of Wemlin Hub](https://reader036.vdocument.in/reader036/viewer/2022070321/558e46c91a28ab51268b46c2/html5/thumbnails/26.jpg)
Microkernel Architectural Style
![Page 27: The Architecture of Wemlin Hub](https://reader036.vdocument.in/reader036/viewer/2022070321/558e46c91a28ab51268b46c2/html5/thumbnails/27.jpg)
Pipelines
all components must hook up together via small set of interfaces/contracts
microkernel style
pure (no external dependencies)
has been done successfully many times (httpd, tomcat to name a few)
![Page 28: The Architecture of Wemlin Hub](https://reader036.vdocument.in/reader036/viewer/2022070321/558e46c91a28ab51268b46c2/html5/thumbnails/28.jpg)
Pipelines
sink / tap
filter
transform
aggregate
![Page 29: The Architecture of Wemlin Hub](https://reader036.vdocument.in/reader036/viewer/2022070321/558e46c91a28ab51268b46c2/html5/thumbnails/29.jpg)
tap
filter
transform
aggregate
sink
![Page 30: The Architecture of Wemlin Hub](https://reader036.vdocument.in/reader036/viewer/2022070321/558e46c91a28ab51268b46c2/html5/thumbnails/30.jpg)
tap
sink
![Page 31: The Architecture of Wemlin Hub](https://reader036.vdocument.in/reader036/viewer/2022070321/558e46c91a28ab51268b46c2/html5/thumbnails/31.jpg)
filter!public interface Filter { ! boolean accept(Object obj); !}
stateless
![Page 32: The Architecture of Wemlin Hub](https://reader036.vdocument.in/reader036/viewer/2022070321/558e46c91a28ab51268b46c2/html5/thumbnails/32.jpg)
transform public interface Transformer { ! Object transform(Object original); !}
stateless
![Page 33: The Architecture of Wemlin Hub](https://reader036.vdocument.in/reader036/viewer/2022070321/558e46c91a28ab51268b46c2/html5/thumbnails/33.jpg)
aggregate public interface Aggregator { ! Optional<?> aggregate(Object obj); !}
![Page 34: The Architecture of Wemlin Hub](https://reader036.vdocument.in/reader036/viewer/2022070321/558e46c91a28ab51268b46c2/html5/thumbnails/34.jpg)
CompositionpipeElement = PipelineBuilder.from(gtfsInputJunction()) .transform(new StoppingPlaceResolver()) .filter(new InvalidStopsFilter()) .transform(new UnresolvedLineVehicleTypeAdder()) .transform(new UnresolvedLineNameAdjuster()) .transform(new LineResolver()) .transform( new LineColorsEnricher(...)) .aggregate(new CacheAggregator(cache())) .to(nullSink());
![Page 35: The Architecture of Wemlin Hub](https://reader036.vdocument.in/reader036/viewer/2022070321/558e46c91a28ab51268b46c2/html5/thumbnails/35.jpg)
Model
immutable: model objects are values (changing means a new object)
algebraic: each object identity is defined by it's contents.
pure: in the sense of no external dependencies
![Page 36: The Architecture of Wemlin Hub](https://reader036.vdocument.in/reader036/viewer/2022070321/558e46c91a28ab51268b46c2/html5/thumbnails/36.jpg)
Schedule
Trip
Stop
Station
Line
Projection
StoppingPlace
![Page 37: The Architecture of Wemlin Hub](https://reader036.vdocument.in/reader036/viewer/2022070321/558e46c91a28ab51268b46c2/html5/thumbnails/37.jpg)
ArchitectureModel
Pure Java
Immutable
Algebraic
Inverse References
!
Pipeline
Functional Microkernel
Filter (stateless, pure function)
Transformer (stateless, pure function)
Aggregator (stateful, function)
Sink (consumer) / Tap (producer)
![Page 38: The Architecture of Wemlin Hub](https://reader036.vdocument.in/reader036/viewer/2022070321/558e46c91a28ab51268b46c2/html5/thumbnails/38.jpg)
pipeElement = PipelineBuilder.from(gtfsInputJunction()) .transform(new StoppingPlaceResolver()) .filter(new InvalidStopsFilter()) .transform(new UnresolvedLineVehicleTypeAdder()) .transform(new UnresolvedLineNameAdjuster()) .transform(new LineResolver()) .transform( new LineColorsEnricher(...)) .aggregate(new CacheAggregator(cache())) .to(nullSink());
![Page 39: The Architecture of Wemlin Hub](https://reader036.vdocument.in/reader036/viewer/2022070321/558e46c91a28ab51268b46c2/html5/thumbnails/39.jpg)
![Page 40: The Architecture of Wemlin Hub](https://reader036.vdocument.in/reader036/viewer/2022070321/558e46c91a28ab51268b46c2/html5/thumbnails/40.jpg)
pipeline “compile” assembly
![Page 41: The Architecture of Wemlin Hub](https://reader036.vdocument.in/reader036/viewer/2022070321/558e46c91a28ab51268b46c2/html5/thumbnails/41.jpg)
![Page 42: The Architecture of Wemlin Hub](https://reader036.vdocument.in/reader036/viewer/2022070321/558e46c91a28ab51268b46c2/html5/thumbnails/42.jpg)
Assemblies
Spring Integration based
Thread-pool based
Fork-join based
![Page 43: The Architecture of Wemlin Hub](https://reader036.vdocument.in/reader036/viewer/2022070321/558e46c91a28ab51268b46c2/html5/thumbnails/43.jpg)
Segments
parallel / serial
separated by queues
segmentation based on the used interfaces
![Page 44: The Architecture of Wemlin Hub](https://reader036.vdocument.in/reader036/viewer/2022070321/558e46c91a28ab51268b46c2/html5/thumbnails/44.jpg)
Fork-Join
to be effective, one must batch
batching introduces latency (in RT)
trick
batch mode
RT mode
![Page 45: The Architecture of Wemlin Hub](https://reader036.vdocument.in/reader036/viewer/2022070321/558e46c91a28ab51268b46c2/html5/thumbnails/45.jpg)
Fork-Join
RT mode
batch mode
![Page 46: The Architecture of Wemlin Hub](https://reader036.vdocument.in/reader036/viewer/2022070321/558e46c91a28ab51268b46c2/html5/thumbnails/46.jpg)
Memoization at construction time
Problem: immutable objects —> lots of created objects
!
@DesignatedFactoryMethod
AspectJ runtime weaver
![Page 47: The Architecture of Wemlin Hub](https://reader036.vdocument.in/reader036/viewer/2022070321/558e46c91a28ab51268b46c2/html5/thumbnails/47.jpg)
hornetbackends frontend
![Page 48: The Architecture of Wemlin Hub](https://reader036.vdocument.in/reader036/viewer/2022070321/558e46c91a28ab51268b46c2/html5/thumbnails/48.jpg)
Scaling
![Page 49: The Architecture of Wemlin Hub](https://reader036.vdocument.in/reader036/viewer/2022070321/558e46c91a28ab51268b46c2/html5/thumbnails/49.jpg)
Storage
In-memory custom store
replacable
just an Aggregator
makes blue-green deployments a beeze
![Page 50: The Architecture of Wemlin Hub](https://reader036.vdocument.in/reader036/viewer/2022070321/558e46c91a28ab51268b46c2/html5/thumbnails/50.jpg)
WinsConstraint
immutable model
algebraic model
Inverse references
controlled state
(functional) microkernel style !
Gain
execution stragety freedom
parallelism
scalability
memoization at constructor time
cacheability
composability
![Page 51: The Architecture of Wemlin Hub](https://reader036.vdocument.in/reader036/viewer/2022070321/558e46c91a28ab51268b46c2/html5/thumbnails/51.jpg)
WinsConstraint
immutable model
algebraic model
Inverse references
controlled state
(functional) microkernel style !
Gain
execution stragety freedom
parallelism
scalability
memoization at constructor time
cacheability
composability
![Page 52: The Architecture of Wemlin Hub](https://reader036.vdocument.in/reader036/viewer/2022070321/558e46c91a28ab51268b46c2/html5/thumbnails/52.jpg)
WinsConstraint
immutable model
algebraic model
Inverse references
controlled state
(functional) microkernel style !
Gain
execution stragety freedom
parallelism
scalability
memoization at constructor time
cacheability
composability
![Page 53: The Architecture of Wemlin Hub](https://reader036.vdocument.in/reader036/viewer/2022070321/558e46c91a28ab51268b46c2/html5/thumbnails/53.jpg)
WinsConstraint
immutable model
algebraic model
Inverse references
controlled state
(functional) microkernel style !
Gain
execution stragety freedom
parallelism
scalability
memoization at constructor time
cacheability
composability
![Page 54: The Architecture of Wemlin Hub](https://reader036.vdocument.in/reader036/viewer/2022070321/558e46c91a28ab51268b46c2/html5/thumbnails/54.jpg)
WinsConstraint
immutable model
algebraic model
Inverse references
controlled state
(functional) microkernel style !
Gain
execution stragety freedom
parallelism
scalability
memoization at constructor time
cacheability
composability
![Page 55: The Architecture of Wemlin Hub](https://reader036.vdocument.in/reader036/viewer/2022070321/558e46c91a28ab51268b46c2/html5/thumbnails/55.jpg)
WinsConstraint
immutable model
algebraic model
Inverse references
controlled state
(functional) microkernel style !
Gain
execution stragety freedom
parallelism
scalability
memoization at constructor time
cacheability
composability
![Page 56: The Architecture of Wemlin Hub](https://reader036.vdocument.in/reader036/viewer/2022070321/558e46c91a28ab51268b46c2/html5/thumbnails/56.jpg)
Architecture is Important
It’s the set of constraints you choose for your system
It how those constraints work in concert
It happens on a smaller scale than you usually think
![Page 57: The Architecture of Wemlin Hub](https://reader036.vdocument.in/reader036/viewer/2022070321/558e46c91a28ab51268b46c2/html5/thumbnails/57.jpg)
AcknowledgmentsClojure (especially Rich Hickey’s talk on values, state and identity)
Laminahttps://github.com/ztellman/lamina
Apache Stormhttps://storm.incubator.apache.org
Casadinghttp://www.cascading.org
Akka http://akka.io
![Page 58: The Architecture of Wemlin Hub](https://reader036.vdocument.in/reader036/viewer/2022070321/558e46c91a28ab51268b46c2/html5/thumbnails/58.jpg)
Clojure Transducers