big data concepts and patterns - inria · thinking big big data concepts and patterns l . p dr....

29
Thinking BIG Big Data Concepts and Patterns l . p Dr. Lilia Sfaxi Data Science Wokshop, ENIT September, 13 th 2017

Upload: others

Post on 21-May-2020

7 views

Category:

Documents


1 download

TRANSCRIPT

Page 1: Big Data Concepts and Patterns - Inria · Thinking BIG Big Data Concepts and Patterns l . p Dr. Lilia Sfaxi Data Science Wokshop , ENIT September, 13 th 2017. DATA 2. DATA ... Big

Thinking BIGBig Data Concepts and

Patterns

l . p

Dr. Lilia SfaxiData Science Wokshop, ENIT

September, 13th 2017

Page 2: Big Data Concepts and Patterns - Inria · Thinking BIG Big Data Concepts and Patterns l . p Dr. Lilia Sfaxi Data Science Wokshop , ENIT September, 13 th 2017. DATA 2. DATA ... Big

DATA2

Page 3: Big Data Concepts and Patterns - Inria · Thinking BIG Big Data Concepts and Patterns l . p Dr. Lilia Sfaxi Data Science Wokshop , ENIT September, 13 th 2017. DATA 2. DATA ... Big

DATAData Mining

DataViz Data Analytics

Open Data

Data Science

Cloud Mobile IoT BI

Big Data

3

Page 4: Big Data Concepts and Patterns - Inria · Thinking BIG Big Data Concepts and Patterns l . p Dr. Lilia Sfaxi Data Science Wokshop , ENIT September, 13 th 2017. DATA 2. DATA ... Big

CollectionProcessing

Storage

Visualization

Security - Monitoring

Business

Data Issues

4

Page 5: Big Data Concepts and Patterns - Inria · Thinking BIG Big Data Concepts and Patterns l . p Dr. Lilia Sfaxi Data Science Wokshop , ENIT September, 13 th 2017. DATA 2. DATA ... Big

Data Issues

Available Data Value Extraction On-Time5

Page 6: Big Data Concepts and Patterns - Inria · Thinking BIG Big Data Concepts and Patterns l . p Dr. Lilia Sfaxi Data Science Wokshop , ENIT September, 13 th 2017. DATA 2. DATA ... Big

Data Issues

Infrastructure

Page 7: Big Data Concepts and Patterns - Inria · Thinking BIG Big Data Concepts and Patterns l . p Dr. Lilia Sfaxi Data Science Wokshop , ENIT September, 13 th 2017. DATA 2. DATA ... Big

Data Issues

Infrastructure

Scalable

Available

Flexible

Volume

Velocity

Variety 7

Page 8: Big Data Concepts and Patterns - Inria · Thinking BIG Big Data Concepts and Patterns l . p Dr. Lilia Sfaxi Data Science Wokshop , ENIT September, 13 th 2017. DATA 2. DATA ... Big

Data Issues

3V 8

Page 9: Big Data Concepts and Patterns - Inria · Thinking BIG Big Data Concepts and Patterns l . p Dr. Lilia Sfaxi Data Science Wokshop , ENIT September, 13 th 2017. DATA 2. DATA ... Big

Store THEN Process

Big Data Principles

Data-driven Decisions (NOT Decision-Driven Data)

Redunduncy is GOOD

There is NO NEEDLESS Data

GO Polyglot!9

Page 10: Big Data Concepts and Patterns - Inria · Thinking BIG Big Data Concepts and Patterns l . p Dr. Lilia Sfaxi Data Science Wokshop , ENIT September, 13 th 2017. DATA 2. DATA ... Big

ScalabilityScale UP vs Scale OUT

10

Classical : Scale UP

Page 11: Big Data Concepts and Patterns - Inria · Thinking BIG Big Data Concepts and Patterns l . p Dr. Lilia Sfaxi Data Science Wokshop , ENIT September, 13 th 2017. DATA 2. DATA ... Big

ScalabilityScale UP vs Scale OUT

11

Big Data : Scale OUT

Page 12: Big Data Concepts and Patterns - Inria · Thinking BIG Big Data Concepts and Patterns l . p Dr. Lilia Sfaxi Data Science Wokshop , ENIT September, 13 th 2017. DATA 2. DATA ... Big

Ring Master- Slave12

ScalabilityArchitectures

Page 13: Big Data Concepts and Patterns - Inria · Thinking BIG Big Data Concepts and Patterns l . p Dr. Lilia Sfaxi Data Science Wokshop , ENIT September, 13 th 2017. DATA 2. DATA ... Big

Classical Architectures Big Data Architecture

Database Application Server

13

ScalabilityCo-Localization of Processing & Storage

Page 14: Big Data Concepts and Patterns - Inria · Thinking BIG Big Data Concepts and Patterns l . p Dr. Lilia Sfaxi Data Science Wokshop , ENIT September, 13 th 2017. DATA 2. DATA ... Big

14

ScalabilityFault Tolerance

Data Replication

Page 15: Big Data Concepts and Patterns - Inria · Thinking BIG Big Data Concepts and Patterns l . p Dr. Lilia Sfaxi Data Science Wokshop , ENIT September, 13 th 2017. DATA 2. DATA ... Big

15

ScalabilityFault Tolerance

Data ReplicationCluster Replication

Page 16: Big Data Concepts and Patterns - Inria · Thinking BIG Big Data Concepts and Patterns l . p Dr. Lilia Sfaxi Data Science Wokshop , ENIT September, 13 th 2017. DATA 2. DATA ... Big

16

ScalabilityFault Tolerance

Data ReplicationCluster ReplicationRack Awareness

Page 17: Big Data Concepts and Patterns - Inria · Thinking BIG Big Data Concepts and Patterns l . p Dr. Lilia Sfaxi Data Science Wokshop , ENIT September, 13 th 2017. DATA 2. DATA ... Big

AvailabilityCAP Theorem

17

Consistency

Availability Partition Tolerance

Pick Any Two

Page 18: Big Data Concepts and Patterns - Inria · Thinking BIG Big Data Concepts and Patterns l . p Dr. Lilia Sfaxi Data Science Wokshop , ENIT September, 13 th 2017. DATA 2. DATA ... Big

AvailabilityCAP Theorem

18

Consistency

Atomicity

Isolation

Durability

Basically Available

Soft-State

Eventual Consistency

ACID BASE

Page 19: Big Data Concepts and Patterns - Inria · Thinking BIG Big Data Concepts and Patterns l . p Dr. Lilia Sfaxi Data Science Wokshop , ENIT September, 13 th 2017. DATA 2. DATA ... Big

AvailabilityTime

19

Stream Processing Support

Dynamic and Interactive Charts and Reports

In-Memory Processing

In-Memory Storage

Page 20: Big Data Concepts and Patterns - Inria · Thinking BIG Big Data Concepts and Patterns l . p Dr. Lilia Sfaxi Data Science Wokshop , ENIT September, 13 th 2017. DATA 2. DATA ... Big

FlexibilityONE application can support…

20

Diverse Data Sources

Schema-less Data

Multiple Processing Paradigms

Multiple Storage Systems

Page 21: Big Data Concepts and Patterns - Inria · Thinking BIG Big Data Concepts and Patterns l . p Dr. Lilia Sfaxi Data Science Wokshop , ENIT September, 13 th 2017. DATA 2. DATA ... Big

Research Domains

21

In the Big Data DomainEVERYTHING

Is Yet to Be Done

Page 22: Big Data Concepts and Patterns - Inria · Thinking BIG Big Data Concepts and Patterns l . p Dr. Lilia Sfaxi Data Science Wokshop , ENIT September, 13 th 2017. DATA 2. DATA ... Big

Research DomainsOptimization

22

Processing Time Optimization

Storage Size and Compression

Data Access Optimization

Tradeoff bw Consistency and Availability

Page 23: Big Data Concepts and Patterns - Inria · Thinking BIG Big Data Concepts and Patterns l . p Dr. Lilia Sfaxi Data Science Wokshop , ENIT September, 13 th 2017. DATA 2. DATA ... Big

Research DomainsData Science

23

Distributed Algorithms for Machine Learning

Semantic & Sentiment Analysis

Visualization Algorithms

Data Mining, Data Prediction, Data Analytics

Page 24: Big Data Concepts and Patterns - Inria · Thinking BIG Big Data Concepts and Patterns l . p Dr. Lilia Sfaxi Data Science Wokshop , ENIT September, 13 th 2017. DATA 2. DATA ... Big

Research DomainsBig Data Design

24

Design Methodologies for Big Data Systems

Standardization of Big Data Architectures

Design and Architectural Patterns

Modeling Language(s) for Schema-less Data

Page 25: Big Data Concepts and Patterns - Inria · Thinking BIG Big Data Concepts and Patterns l . p Dr. Lilia Sfaxi Data Science Wokshop , ENIT September, 13 th 2017. DATA 2. DATA ... Big

Research DomainsBig Data Security

25

Non-Relational Databases Security

Logs Gathering and Analysis

Source Data Validation and Filtering

Access Control and Cryptography

Page 26: Big Data Concepts and Patterns - Inria · Thinking BIG Big Data Concepts and Patterns l . p Dr. Lilia Sfaxi Data Science Wokshop , ENIT September, 13 th 2017. DATA 2. DATA ... Big

Research DomainsBig Data & OtherTrends

26

Big Data & Business Intelligence

Big Data & Cloud Computing

Big Data & Internet of Things

Big Data & Mobile

Page 27: Big Data Concepts and Patterns - Inria · Thinking BIG Big Data Concepts and Patterns l . p Dr. Lilia Sfaxi Data Science Wokshop , ENIT September, 13 th 2017. DATA 2. DATA ... Big

Research DomainsBig Data & Business Fields

27

Big Data in Education

Big Data in Health

Big Data in Art

Big Data in Finance

Page 28: Big Data Concepts and Patterns - Inria · Thinking BIG Big Data Concepts and Patterns l . p Dr. Lilia Sfaxi Data Science Wokshop , ENIT September, 13 th 2017. DATA 2. DATA ... Big

In Other Words…

28

Big Data is Here to STAY!

Page 29: Big Data Concepts and Patterns - Inria · Thinking BIG Big Data Concepts and Patterns l . p Dr. Lilia Sfaxi Data Science Wokshop , ENIT September, 13 th 2017. DATA 2. DATA ... Big

Thinking BIGBig Data Concepts and

Patterns

l . p

Dr. Lilia SfaxiApril 2017