taming big data with modern software architecture

Software Architecture

Taming Big Data With

Modern Software Architecture Big Data User Group

Stuttgart, 26.09.2013

Thomas Widmann Managing Director, WidasConcepts

Vinoth Kannan Senior Consultant, WidasConcepts

About the Speakers

2

Thomas Widmann

Vinoth Kannan !   Senior Consultant in WidasConcepts !   Working primarily around Big Data ecosystems of Hadoop and Storm !   Experienced in IT industries of Europe, North America and Asia !   Masters degrees from Politecnico di Torino, Italy & Grenoble INP, France ! Contact: [email protected]

!   Managing Director of WidasConcepts !   Expert in modern Sales- and Service-Solutions !   More than two decades of consulting experience in Web X.0

concepts, continuous delivery and IT-strategies !   Big Data Evangelist ! Contact: [email protected]

Who we are

!   Innovative IT-Consulting Company founded in 1997

! Headquaters in Wimsheim with offices in Munich and Essen

! Specialized in Software for Modern Business in the areas of !   Strategic IT Consulting !   Next Business Intelligence !   Software Development

!   35 expert consultants specializing in !   Data Science !   Software Architecture !   Mobile & Web Development !   Business Consulting

WidasConcepts GmbH

3

Agenda

4

How Big Data can help business

Big Data applied in CarbookPlus Platform

New Software Architectures for Big Data

Q & A

How big data can help business

5

Big challenges of todays business

eCommerce

Financial Services

Telecommunication

HealthCare

Marketing

National Security

o  Increase in Sales o  Determine pricing o  Avoid Shop cart abandoners o  Manage logistics and inventory

o  Fraud detection o  Real Time analytics on

financial data o  Managing mobile payments

o  Manage network traffic overload

o  Hardware cost optimisation based on traffic data

o  Real time monitoring of patients vitals for predictive intervention and care.

o  Cost savings on patient care

o  Mood prediction for market research

o  Real time bidding o  Location based marketing

o  Monitor real time feed from CCTV, traffic data, mobile data

o  Find patterns/cluster for people, places, products, things

Building sector, all manufacturers, energy, media and many more

Getting the Right Data to the Right People at the Right Time

6

Mobile Payment

Money Cards / POS

Banking Portal

Adviser / Agency

Banking Operations

Event Database

Business Activity

Monitoring

CEP

Making Use of Events

Real Time Analysis

Cross-Up-Selling

Fraud Detection

Product Innovation

Impact of Big Data on Business Results on Europe-wide survey by TU-Berlin, Sept 2013

Source: European Smart Data Survey, Sept 2013

7

Need for Big Data 4Vs concept of Big Data

Volume

To handle the huge amount of data genereted

by businesses

Velocity

To store, analyse and retrive huge dataset in

good speed

Variety To process data from

multiple sources, majorly unstructured data

Source: Oracle

Value To ask the right questions

to generate maximum value

8

What makes data so big? Combination of structured, unstructured data created by sensors, actors, humans

9

New Software Architecture Need of the hour for modern architecture

analyse huge

volume of data

fast

consolidate data from mul;ple sources

Intelligent

predic;ve

10

Agenda

11



New Software architectures for Big Data

Q & A

Datawarehouse – Classic BI

Online System

Database n

Classic Software Architectures

Main goal: keep business records

<Client> Browser

Presentation

Business Logic

Database 2

Staging ETL ETL

BI Reporting

Database 1

Batch Processing

Main goal: Reporting

12

Need for real time system for enterprise

13

The Approach

Impact of analytics, event processing and decision management in building a real time big data system for enterprise

Acquire Data: •  Business Trx./Events •  Observational Events

from anywhere

Observe Orient Decide Act

Systems: •  Various systems •  Preferrable: MQ

•  put in context •  Evaluate data, predict •  determine actions •  Initiate actions •  manage data

Systems: •  complex event processing,

rule management •  realtime analysis with new

big data technologies

•  select action, if necessary

•  Perform action (manually or automated)

Systems: •  Business software •  Decision support

software •  Reporting

(John Boyd) OODA-Loop ist ein Informationsstrategiekonzept aus dem militärischen Bereich

Phases of realising real time system Step by step approach for the architecture

• Create structured data from raw data • Extract, merge, filter etc Data Extraction

• Aggregate and select relevant data • Speed and flexibility Model development

• Testing the model • Comparing the result Evaluation &

deployment

• Triggered by action in the decision layer • Score to validate output Real-time Scoring

• Need to refresh the ever changing data • Real time data view Model Refresh

14

Big Data Architecture

Service Layer CQRS

Distributed DB

Distributed Messaging System

Recommendations

Realtime-Processing Batch-Processing Lambda architecture

Business Transactional Data Business Analytics Data Recommendation Data

Real Time Event Alert Data

Big Data Eco System

Distributed Search Engine

Business Data Formats

Contextual Data Business Transactions

Reporting Notifications

Notification System Data Retrieval

In-Memory Technology

15

Distributed DB NoSQL DB

Basic idea behind Lambda Architecture

query = function(all data) Nathan Marz

Big Data - principles and best practices of scalable realtime data systems

16

Basic Idea behind Lambda

𝑓(𝑎↓0 …𝑎↓𝑚 )

Perform some function from real-time data “0“ to the history data “n“

Real Time Big Data

𝑓(𝑎↓0 .. 𝑎↓𝑛 …𝑎↓𝑚 ) = 𝑓(𝑎↓0 .. 𝑎↓𝑛 ) + 𝑓(𝑎↓𝑛+1 …𝑎↓𝑚 )

Lambda Architecture

Hadoop Process Storm Process

Real Time Big Data

} } }

Letting the History data processed by Hadoop makes process faster

17

Agenda

18




Q & A

What is Carbookplus? Change the way you drive.

19

Why Carbookplus? Benefits of being a Carbookplus member

Track trips Maintain Logbook Get Notified about best gas stations & snipers Manage and compare vehicle cost Fleet management Social platform connecting drivers

Big Data in Carbookplus Where do we use it

Trip Transac;on Data

Recommend possible trips Create trips Update logbook entry No;fy user for discrepancy in the logbook entry No;fy user for best gas sta;on Update vehicle mileage Generate analy;c reports

Real Time !!!

Big Data in Carbookplus Event Driven Model

Use Case : Track trips on Real Time

Start address

Intermediate address 1

End address

Event 1

Event 2

Event n Intermediate address n

…..

Event n+1

Object Model

Event …

Big Data in Carbookplus How do we use it

Trip Transac;on Data

Real Time Storm Cluster

In Memory Data Processing

MongoDB

Trip Data Hadoop Cluster

Map & Reduce

Analy;cal Data Decision Management

Recommender Engine

Rule Engine

KaYa

ETL

Output Mobile Web BI

User Data ETL

Vehicle Data ETL

Historical Data

Real Time Data

Conclusion

Ø  Big Data clearly improves business efficiency and answers lot of the questions which were previously unanswerable in this data driven world.

Ø  Need to adapt our IT architecture to remain competitive and see the big picture

Ø  Flipside : No standardised implementation architecture. Need to be tailor-made for individual business needs.

Ø  Road Ahead : Need for phased integration into Big Data ecosystem Ø  We are hiring !!

§  Data Scientists §  Big Data Engineers §  Interns

Visit us : www.widas.de Email : [email protected]

Agenda

25

How BigData can help business

BigData applied in CarbookPlus Platform


Q & A