pivotal digital transformation forum: journey to become a data-driven enterprise

Journey To Becoming a Data-Driven Enterprise:

Pivotal Big Data Suite Technical Overview

Feras Alsamawi Senior Field Engineer Pivotal

2 © Copyright 2015 Pivotal. All rights reserved.

Agenda

•  Big Data Challenges

•  The Value of Data

•  Pivotal Big Data Suite

•  The Open Data Platform

•  Big Data Architectures

© Copyright 2015 Pivotal. All rights reserved.

THE POWER OF 1

R X

Increasing Freight Utilization Rail

Predictive Maintenance Healthcare

Predictive Diagnostics Power

Driving Outcomes That Matter

One Percent Improvement Equals

$27B Industry Value by Reducing System

Inefficiency

$63B Industry Value by Reducing Process

Inefficiency

$66B Industry Value with

Efficiency Improvements In Gas-fired Power

Plant Fleets Source: General Electric


BIG DATA CHASM

70% of data

generated by customers

80% of data stored

3% prepared for

analysis

0.5% being

analyzed

<0.5% being

operationalized

4

THE DATA DIVIDE


Software Is Eating The World

Data Is Fueling Software

SOFTWARE IS EATING THE WORLD


JOURNEY TO A DATA-DRIVEN ENTERPRISE

Deploy analytic apps and automate at scale

Perform advanced analytics Discover insights

Modernize data infrastructure


The value of Data

Time

Value of Information

µs ms s hour day month year yr+


Traditional Systems

The value of Data

Time




Speed Layer

Traditional Systems

Pivotal’s λ Architecture

Serving Layer “Big Data”

Time



Spring XD

Pivotal HD

Pivotal GemFire

Batch Layer


WHY PIVOTAL FOR BIG DATA ? Complete platform

SQL on Hadoop leadership

Deployment options

Open source

Flexible licensing

Advanced data services

Pivotal Data Engineering Pivotal Labs Pivotal Data Science

11 © Copyright 2015 Pivotal. All rights reserved. 11 © Copyright 2013 Pivotal. All rights reserved.

Big Data Suite Components


HDFS Data Lake Expert System /

Machine Learning

In-Memory Real-Time Data

Continuous Learning Continuous Improvement

Continuous Adapting

Data Stream Pipeline

Multiple Data Sources Real-Time Processing Store Everything

Pro-Active, Self-Improving, Machine Learning Systems


Ingest Transform Sink

SpringXD GemFire

Data Stream Needs an Agile, Scalable and Fast Solution

HAWQ GPDB

Data Lake



SpringXD

Distributed Computing

In-Memory Real-Time Data

Spring XD Orchestrates and Automates all the Steps on Data Stream Pipelining

Expert System / Machine Learning

Extensible Open-Source Fault-Tolerant Horizontally Scalable HAWQ GPDB

Data Lake


Ingest / SINK Process Analyze

•  No coding required

•  Dozens of built-in connectors

•  Seamless integration with Kafka, Sqoop

•  Create new connectors easily using Spring

•  Call Spark, Reactor or RxJava

•  Built-in configurable filtering, splitting and transformation

•  Out-of-box configurable jobs for batch processing

•  Import and invoke PMML jobs easily

•  Call Python, R, Madlib and other tools

•  Built-in configurable counters and gauges

Spring XD State of the Art Data Pipeline Automation



SpringXD


GemFire Provides Scalable, Low-Latency Data Access, Storage and Event Processing

Expert System / Machine Learning

GemFire

Extensible Open-Source Fault-Tolerant Horizontally Scalable HAWQ GPDB

Data Lake


GemFire

•  In-Memory Enterprise Data Grid •  Horizontally Scalable, Consistent, Highly

Available

•  Event handling •  Continuous Queries •  Enterprise Data Geo Distribution

In-memory Real Time Data



SpringXD


Pivotal Provides SQL Based Advanced Analytics

GemFire

Extensible Open-Source Fault-Tolerant Horizontally Scalable

Data Lake

HAWQ GPDB


HAWQ

•  Massively Parallel Processing RDBMS on HADOOP

•  ANSI SQL on Hadoop •  Extremely high performance for

analytics (not like Hive) •  Stores all data directly on

HDFS

•  Functions in MADlib, R/Python/Java, Perl, pgsql or C languages

Advanced SQL analytics in Hadoop

Combining SQL with Hadoop is key for analytics

SQL remains #1 choice for Data Science


Data Streaming Reference Architecture Data Feeds Transactional Apps Analytic Apps


Distributed Computing Real-Time Data

Expert Systems & Machine Learning

Advanced Analytics

HDFS Data Lake


Data Streaming Reference Architecture Data Feeds Transactional Apps Analytic Apps


HDFS Data Lake

GemFire HAWQ GPDB

SpringXD

BUILT FOR THE SPEED OF BUSINESS


Financial Compliance

BUSINESS PROBLEM •  Ensure compliance with Dodd-Frank and Basel

Committee regulations

•  Identify underlying risk and fraud while reducing the compliance department’s overburdened

Emails Chats Trades

Transactions Policy Securities

Phone Calls Watch Lists …

Financial compliance Data Lake

Data integration

Data clean up Modeling

Classification and ranking

Analyst user interfaces Feedback

Analytics

Analyst feedback Data integration: e.g., append trade information with email and chat communications

Data cleanup: e.g., identify newsletters and spam emails

Modeling: •  Predictive modeling to flag

messages and trades •  Graph and cohort analysis

Analyst feedback Reviewed fraud instances included in periodic model refreshes

SOLUTION �  A data lake platform coupled with cutting edge data

science techniques

�  Flexible user interface to promote an adaptive, continuously learning compliance framework


Pivotal Topic & Sentiment Analysis Engine

External Tables

PXF

HDFS

Source: http Sink: hdfs

Parallel Parsing of JSON

(PL/Python)

HAWQ

Nightly Cron Jobs

Topic Analysis through MADlib pLDA

Unsupervised Sentiment Analysis

(PL/Python)

D3.js

Spring XD

Twitter Decahose (~55 million tweets/day)


The Connected Car Architecture INGESTION

JSON / HTTP

STREAM PROCESSING

Spring XD Transform Enrich

DATA LAKE

Pivotal HD Sink

ADVANCED ANALYTICS

HAWQ

REAL-TIME DATA INSIGHTS

GemFire

MOBILE SERVICES

MICROSERVICES

Pivotal CF Dashboard Analytics App Simulator

IoT APPS

Rabbit MQ

PUSH


Horizontally Scalable Fault Tolerant Extensible Open-Source

STREAM PROCESSING

Spring XD

Rabbit MQ

DATA LAKE

Pivotal HD

ADVANCED ANALYTICS

HAWQ

ENRICHER PREDICTIVE ANALYTICS

+ Timestamp & GUID

+ MPG, rangE & route

MOBILE APP

JSON

REAL-TIME DATA INSIGHTS

GemFire

CAR SENSOR

Sink

Tap

DASHBOARD


FOR FURTHER INFO…

•  Pivotal Data Product Info, Docs and Downloads @ http://pivotal.io/big-data

•  Pivotal Blog @ http://blog.pivotal.io

•  Pivotal Data Science Blog @ http://blog.pivotal.io/data-science-pivotal

•  Pivotal Academy @ https://pivotal.biglms.com

•  Or reach out to your local Pivotal Account Executive…

Digital Transformation Forum

Disrupt or Be Disrupted 19 OCTOBER · BMW WELT EVENT CENTRE · MUNICH

pivotal digital transformation forum: journey to become a data-driven enterprise

Data & Analytics