omnia in 20 minutes
Post on 11-Feb-2017
468 Views
Preview:
TRANSCRIPT
Presented by Daniel Ortiz EsquivelR&D Software EngineerYAPC::Europe September 2015
Site: https://developer.williamhill.comTwitter: https://twitter.com/dortizesquivel
OMNIADistributed & Reactive
platform for data management
Motivations
2Omnia: Distributed & Reactive platform for data management
Users
Feeds
System
3 Party
In order to be in a position to innovate we need to control and understand our data
SocialNetwork
s
IoT
William Hill
DMP based on the Lambda architecture and the Reactive principles
What is Omnia?
3
Chr
onos
Dat
a S
ourc
e NeoCortexSpeed Layer
FatesBatch Layer
Her
mes
Ser
ving
Lay
er
Data Flow
Input Output
Omnia: Distributed & Reactive platform for data management
Resources:• Reactive Extension for Perl:
https://github.com/eilara/Rx.pl• Carl Mäsak YAPC:2013: https
://www.youtube.com/watch?v=EnlYURSKgk0
Reactive principles
4
Responsive
Resilient
Message Driven
Elastic
The Reactive Manifestohttp://www.reactivemanifesto.org/
Omnia: Distributed & Reactive platform for data management
Chronos is a reliable and scalable component which collect data from different source and organize them into Streams of observable incidents.
Chronos: Data acquisition
5
Incident: { type: “bet”, version: “1.0”, time: “2015-09-03 06:00:10”, acquisitionTime: “2015-09-03 06:00:06”, source: “BetSystem”, payload: {…. Any valid JSON}}
Omnia: Distributed & Reactive platform for data management
Chr
onos
Dat
a S
ourc
e
…
JMS
HTTP Poll
SSE
Adapter
Streams
Converter Persistence
Bets
Depo
sits
Pric
es Stream = Adapter + Converter + Persistence
Chronos: Data acquisition
6Omnia: Distributed & Reactive platform for data management
Chronos 1(SSE, Bets placed)
Chronos 2(JMS, Deposits)
Chronos 3(HTTP, Events)
Chronos N(SSE, Twitter)
….…
Chronos 2(JMS, Deposits)
(SSE, Bet Placed)
Perl Consumer & Producer Module: http://search.cpan.org/~sgladkov/Kafka/lib/Kafka.pm
Fates represents the long term memory of Omnia. It organizes the incidents that Chronos collected into timelines and also elaborates new information as views by using machine learning, logical reasoning and time series analysis.
Fates: Batch layer
7Omnia: Distributed & Reactive platform for data management
Customer: 123
Login
Deposit
Bet placed
…Logout
Event: 78
Started
Fault
Penalty
…
GoalTimelines & Views
Bets Deposits Events Session
FatesBatch Layer
Fates: Batch layer
8Omnia: Distributed & Reactive platform for data management
Timelines
Views
Jobs
Fates
Perl Module for Cassandra: http://search.cpan.org/~mkjellman/perlcassa-0.041/lib/perlcassa.pm
NeoCortex represents the short term memory of Omnia. It offers a framework to develop micro services on top of Apache Spark. It performs fast and real time data processing with the data acquired by Chronos.
NeoCortex: Speed layer
9Omnia: Distributed & Reactive platform for data management
NeoCortex
Bets
Depo
sits
Eve
nts
Sess
ion
Micro Services
Out
put
Hermes is a scalable and full duplex communication for B2C and B2B.
Hermes: Serving Layer
10
Omnia: Distributed & Reactive platform for data management
B2CBrowser
B2B
Load
bal
ance
r
Push Server
DistributeCache Push
Server
Push Server
…
TCP
WS
HTTP
JS A
PI
WH AppsC
ache
Cac
he
Apps
Custom advert, bonus, data load prediction, bot detection...
Omnia Inception
11
Chr
onos
Dat
a S
ourc
e NeoCortexSpeed Layer
FatesBatch Layer
Her
mes
Ser
ving
Lay
er
Input Output
Omnia: Distributed & Reactive platform for data management
Users become a new data producer
Real time monitoring and elasticityDocker and Mesos: Scale In&Out based on demand
Omnia inception +1
12
Chr
onos
Dat
a S
ourc
e NeoCortexSpeed Layer
FatesBatch Layer
Her
mes
Ser
ving
Lay
er
Input Output
Omnia: Distributed & Reactive platform for data management
JMX
JMX
JMX
Omnia infrastructure
13
Omnia: Distributed & Reactive platform for data management
Omnia
Docker
Marathon
Mesos
Node Node NodeNodeNode
https://careers.williamhillplc.com/
Questions
14
Omnia: Distributed & Reactive platform for data management
Omnia will be released as an open source project
top related