DevOps at ING one Analytics

Giuseppe d’Alessio

Combining data engineering with data operations

Moscow, 16 November 2017

Bio• Made in Italy

• Living in the Netherlands

• SW Engineering, Machine Learning & Pattern Recognition

Work• Data Engineer @ ING Fast Data NL• Analytics @ ING Direct Italy

• Digital Channels / Security @ ING WBS NL

Giuseppe d’Alessio

https://nl.linkedin.com/in/giuseppedalessio@peppeweb

“Поехали! Poechali!” cit.

ING is a top financial enterprise, operating since 1881

Customers

33 MillionPrivate, Corporate and Institutional Customers

Countries

41 In Europe, Asia,

Australia, North and South America

Employees

52,000

Market leaders Benelux

Growth markets

Commercial Banking

Challengers

3

ING’s Think Forward strategy requires a data-driven approach

4

Continuous Analytics

In ING’s one way of working, ‘business’ and ‘IT’ go hand-in-hand

(Biz)DevOps Continuous Delivery

Squad

In ING’s one way of working, ‘business’ and ‘IT’ go hand-in-hand

SquadSquad

Chapter

Guild

DS

DA

OpsOps

CJE

DS

DA

DS

CJE

DS

Ops

DS

CJE

Dev DevDevDevDev

Tribe

AgileCoach

DevOps and Continuous Delivery journey

8

Engineering Culture

ContinuousDelivery

Team commitment

Enterprise commitment

ContinuousDelivery as a Service

2010

Continuous Delivery as a Service (video)

9

Continuous Delivery as a Service

10

The pipeline provides:

• Standardization

• Automation

• Deliver and collect metrics

to improve it.

• Best practices

Release & Acceptance criteria

11

• Toll gate

• Collect all the evidences to assure:

• Code Quality

• SCAT performance

• Security

• Change management process

• IT Risk under control

Actionable Insights

Streaming Analytics enables us to detect patterns in real time, and respond to events for customers’ benefit

13

Secure and reliable

Relevant

Personal

Omnichannel

Predictive Actionable

Credit card payment

15

Travel Insurance

16

Technology

Streaming Data Platform

18

CEP Engine Machine Learning Engine Post-Processor

RawEvent

BusinessEvent

NotificationEvent

“detect pattern”

“determinerelevant

notifications”

“produce notification”

Application Flow :

Kafka Events:

Data storage:

Business Flow:

CustomerProfiles

NotificationDefinitions

Models

GetCustomer

Profile

ApplySelectionCriteria

Score Notificatio

ns

DetectPattern

Get Intermed

Event

Get RawEvents

SendOutput

Business

Users

System configuration: Machine Learning

Environment

ConfigurationGUI

(future scope) Data

Scientists

CreateIntermed

Event

GetBusiness

Event

CreateBusiness

Event

IntermediateEvent

Format Event

Data Lake

Models

Feedback loop

Flink:• is an open source framework for distributed, in-memory (big) data analytics• Likes Java, Scala, and (a bit) Python• Several APIs for streams (DataStream), batch (DataSet), and relational data (Table API)

Benefits:• true streaming: per-event processing, no micro-batching• high volumes, low latency

Features:• state management and fault tolerancy: savepointing, checkpointing exactly-once

semanctics• time windowing: event time, flexible (e.g. sum all transactions in the past minute)• Complex Event Processing, Machine Learning, SQL (at various levels of maturity)

What is Flink, and why use it?

19

Complex Event Processing with FlinkCEP

20

• Allows detection of event patterns in data streams

• In contrast to traditional DBMSs where a query is executed on stored data, CEP executes

data on a stored query.

Conclusions

22

Questions

Follow us to stay a step ahead

ING.com

YouTube.com/ING

SlideShare.net/ING@ING_News LinkedIn.com/company/ING

Flickr.com/INGGroupFacebook.com/ING