omnia in 20 minutes

Presented by Daniel Ortiz EsquivelR&D Software EngineerYAPC::Europe September 2015

Site: https://developer.williamhill.comTwitter: https://twitter.com/dortizesquivel

OMNIADistributed & Reactive

platform for data management

https://developer.williamhill.com/

https://twotter.com/dortizesquivel

Motivations

2Omnia: Distributed & Reactive platform for data management

Users

Feeds

System

3 Party

In order to be in a position to innovate we need to control and understand our data

SocialNetwork

s

IoT

William Hill

DMP based on the Lambda architecture and the Reactive principles

What is Omnia?

3

Chr

onos

Dat

a S

ourc

e NeoCortexSpeed Layer

FatesBatch Layer

Her

mes

Ser

ving

Lay

er

Data Flow

Input Output

Omnia: Distributed & Reactive platform for data management

Resources:• Reactive Extension for Perl:

https://github.com/eilara/Rx.pl• Carl Mäsak YAPC:2013: https

://www.youtube.com/watch?v=EnlYURSKgk0

Reactive principles

4

Responsive

Resilient

Message Driven

Elastic

The Reactive Manifestohttp://www.reactivemanifesto.org/


https://github.com/eilara/Rx.pl

https://github.com/eilara/Rx.pl

https://www.youtube.com/watch?v=EnlYURSKgk0



http://www.reactivemanifesto.org/

http://www.reactivemanifesto.org/

Chronos is a reliable and scalable component which collect data from different source and organize them into Streams of observable incidents.

Chronos: Data acquisition

5

Incident: { type: “bet”, version: “1.0”, time: “2015-09-03 06:00:10”, acquisitionTime: “2015-09-03 06:00:06”, source: “BetSystem”, payload: {…. Any valid JSON}}


Chr

onos

Dat

a S

ourc

e

…

JMS

HTTP Poll

SSE

Adapter

Streams

Converter Persistence

Bets

Depo

sits

Pric

es Stream = Adapter + Converter + Persistence

Chronos: Data acquisition


Chronos 1(SSE, Bets placed)

Chronos 2(JMS, Deposits)

Chronos 3(HTTP, Events)

Chronos N(SSE, Twitter)

….…

Chronos 2(JMS, Deposits)

(SSE, Bet Placed)

Perl Consumer & Producer Module: http://search.cpan.org/~sgladkov/Kafka/lib/Kafka.pm

http://search.cpan.org/~sgladkov/Kafka/lib/Kafka.pm

http://search.cpan.org/~sgladkov/Kafka/lib/Kafka.pm

Fates represents the long term memory of Omnia. It organizes the incidents that Chronos collected into timelines and also elaborates new information as views by using machine learning, logical reasoning and time series analysis.

Fates: Batch layer


Customer: 123

Login

Deposit

Bet placed

…Logout

Event: 78

Started

Fault

Penalty

…

GoalTimelines & Views

Bets Deposits Events Session

FatesBatch Layer

Fates: Batch layer


Timelines

Views

Jobs

Fates

Perl Module for Cassandra: http://search.cpan.org/~mkjellman/perlcassa-0.041/lib/perlcassa.pm

http://search.cpan.org/~mkjellman/perlcassa-0.041/lib/perlcassa.pm



NeoCortex represents the short term memory of Omnia. It offers a framework to develop micro services on top of Apache Spark. It performs fast and real time data processing with the data acquired by Chronos.

NeoCortex: Speed layer


NeoCortex

Bets

Depo

sits

Eve

nts

Sess

ion

Micro Services

Out

put

Hermes is a scalable and full duplex communication for B2C and B2B.

Hermes: Serving Layer

10


B2CBrowser

B2B

Load

bal

ance

r

Push Server

DistributeCache Push

Server

Push Server

…

TCP

WS

HTTP

JS A

PI

WH AppsC

ache

Cac

he

Apps

Custom advert, bonus, data load prediction, bot detection...

Omnia Inception

11

Chr

onos

Dat

a S

ourc


FatesBatch Layer

Her

mes

Ser

ving

Lay

er

Input Output


Users become a new data producer

Real time monitoring and elasticityDocker and Mesos: Scale In&Out based on demand

Omnia inception +1

12

Chr

onos

Dat

a S

ourc


FatesBatch Layer

Her

mes

Ser

ving

Lay

er

Input Output


JMX

JMX

JMX

Omnia infrastructure

13


Omnia

Docker

Marathon

Mesos

Node Node NodeNodeNode

https://careers.williamhillplc.com/

Questions

14


Omnia will be released as an open source project

omnia in 20 minutes

Documents