oracle stream analytics - simplifying stream processing

43
BASEL BERN BRUGG DÜSSELDORF FRANKFURT A.M. FREIBURG I.BR. GENEVA HAMBURG COPENHAGEN LAUSANNE MUNICH STUTTGART VIENNA ZURICH Oracle Stream Analytics Simplifying Stream Processing 29.9.2016 DOAG 2016 Big Data Days Guido Schmutz

Upload: guido-schmutz

Post on 21-Jan-2017

339 views

Category:

Technology


8 download

TRANSCRIPT

Page 1: Oracle Stream Analytics - Simplifying Stream Processing

BASEL BERN BRUGG DÜSSELDORF FRANKFURT A.M. FREIBURG I.BR. GENEVA HAMBURG COPENHAGEN LAUSANNE MUNICH STUTTGART VIENNA ZURICH

Oracle Stream AnalyticsSimplifying Stream Processing

29.9.2016 – DOAG 2016 Big Data DaysGuido Schmutz

Page 2: Oracle Stream Analytics - Simplifying Stream Processing

Guido Schmutz

Working for Trivadis for more than 19 yearsOracle ACE Director for Fusion Middleware and SOACo-Author of different booksConsultant, Trainer, Software Architect for Java, SOA & Big Data / Fast DataMember of Trivadis Architecture BoardTechnology Manager @ Trivadis

More than 25 years of software development experience

Contact: [email protected]: http://guidoschmutz.wordpress.comSlideshare: http://www.slideshare.net/gschmutzTwitter: gschmutz

Oracle Stream Analytics - Simplifying Stream Processing2

Page 3: Oracle Stream Analytics - Simplifying Stream Processing

Unser Unternehmen.

Oracle Stream Analytics - Simplifying Stream Processing3

Trivadis ist führend bei der IT-Beratung, der Systemintegration, dem Solution Engineering und der Erbringung von IT-Services mit Fokussierung auf -und -Technologien in der Schweiz, Deutschland, Österreich und Dänemark. Trivadis erbringt ihre Leistungen aus den strategischen Geschäftsfeldern:

Trivadis Services übernimmt den korrespondierenden Betrieb Ihrer IT Systeme.

B E T R I E B

Page 4: Oracle Stream Analytics - Simplifying Stream Processing

KOPENHAGEN

MÜNCHEN

LAUSANNEBERN

ZÜRICHBRUGG

GENF

HAMBURG

DÜSSELDORF

FRANKFURT

STUTTGART

FREIBURG

BASEL

WIEN

Mit über 600 IT- und Fachexperten bei Ihnen vor Ort.

Oracle Stream Analytics - Simplifying Stream Processing4

14 Trivadis Niederlassungen mitüber 600 Mitarbeitenden.

Über 200 Service Level Agreements.

Mehr als 4'000 Trainingsteilnehmer.

Forschungs- und Entwicklungsbudget: CHF 5.0 Mio.

Finanziell unabhängig undnachhaltig profitabel.

Erfahrung aus mehr als 1'900 Projekten pro Jahr bei über 800 Kunden.

Page 5: Oracle Stream Analytics - Simplifying Stream Processing

Agenda

1. Introduction to Streaming Analytics2. Oracle Stream Analytics3. Demo

Oracle Stream Analytics - Simplifying Stream Processing5

Page 6: Oracle Stream Analytics - Simplifying Stream Processing

Introduction to Streaming Analytics

Oracle Stream Analytics - Simplifying Stream Processing6

Page 7: Oracle Stream Analytics - Simplifying Stream Processing

Traditional Data Processing - Challenges

• Introduces too much “decision latency”

• Responses are delivered “after the fact”

• Maximum value of the identified situation is lost

• Decision are made on old and stale data

• “Data a Rest”

Oracle Stream Analytics - Simplifying Stream Processing7

Page 8: Oracle Stream Analytics - Simplifying Stream Processing

The New Era: Streaming Data Analytics / Fast Data

• Events are analyzed and processed in real-time as the arrive

• Decisions are timely, contextual and based on fresh data

• Decision latency is eliminated

• “Data in motion”

Oracle Stream Analytics - Simplifying Stream Processing8

Page 9: Oracle Stream Analytics - Simplifying Stream Processing

Event / Stream Processing Architecture

DataIngestion

Batchcompute

DataSources

Channel

DataConsumer

Reports

Service

AnalyticTools

AlertingTools

Content

Logfiles

Social

RDBMS

ERP

Sensor

Machine

(Analytical)Real-TimeDataProcessing

Stream/EventProcessing

ResultStore

Messaging

ResultStore

Oracle Stream Analytics - Simplifying Stream Processing

=DatainMotion =DataatRest9

Page 10: Oracle Stream Analytics - Simplifying Stream Processing

“Lambda Architecture” for Big Data

DataIngestion

(Analytical)BatchDataProcessing

Batchcompute

ResultStoreDataSources

Channel

DataConsumer

Reports

Service

AnalyticTools

AlertingTools

Content

RDBMS

Social

ERP

Logfiles

Sensor

Machine

(Analytical)Real-TimeDataProcessing

Stream/EventProcessing

Batchcompute

Messaging

ResultStore

QueryEngine

ResultStore

ComputedInformation

RawData(Reservoir)

Oracle Stream Analytics - Simplifying Stream Processing

=DatainMotion =DataatRest

PullingIngestion

10

Page 11: Oracle Stream Analytics - Simplifying Stream Processing

When to Stream / When not?

Oracle Stream Analytics - Simplifying Stream Processing11

ConstantlowMilliseconds&under

Lowmillisecondstoseconds,delayincaseoffailures

10sofsecondsofmore,Re-runincaseoffailures

Real-Time Near-Real-Time Batch

Page 12: Oracle Stream Analytics - Simplifying Stream Processing

“No free lunch”

Oracle Stream Analytics - Simplifying Stream Processing12

ConstantlowMilliseconds&under

Lowmillisecondstoseconds,delayincaseoffailures

10sofsecondsofmore,Re-runincaseoffailures

Real-Time Near-Real-Time Batch

“Difficult”architectures,lowerlatency “Easierarchitectures”,higherlatency

Page 13: Oracle Stream Analytics - Simplifying Stream Processing

Why Event / Stream Processing?

Oracle Stream Analytics - Simplifying Stream Processing13

Visualize Business in real-time• Dashboards can help people to visualize, monitor and make sense of massive amount of

incoming data in real-time

Detect Urgent Situations• Based on simple or complex analytical patterns of urgent business events• Urgent because they happen in real-time

Automate immediate actions• Run in the background quietly until detecting an urgent situation (risk or opportunity)• Alerts can go to humans through email, text or push notifications or to other applications trough

message queues or service call

Page 14: Oracle Stream Analytics - Simplifying Stream Processing

Oracle Stream Analytics - Simplifying Stream Processing15

Page 15: Oracle Stream Analytics - Simplifying Stream Processing

Oracle Stream Analytics

Oracle Stream Analytics - Simplifying Stream Processing16

Page 16: Oracle Stream Analytics - Simplifying Stream Processing

History of Oracle Stream Analytics

OracleComplexEventProcessing(OCEP)

OracleEventProcessing(OEP)

OracleStreamExplorer(SX)

OracleEventProcessingforJavaEmbedded

OracleStreamAnalytics(OSA)

OracleEdgeAnalytics(OAE)

BEAWeblogic EventServerOracleCQL

OracleIoT CloudService

2016

2015

2007

2008

2012

2013

Oracle Stream Analytics - Simplifying Stream Processing17

Page 17: Oracle Stream Analytics - Simplifying Stream Processing

OEA

• Filtering• Correlation• Aggregation• Pattern

matching

Devices / Gateways

Services

Computing Edge Enterprise

“Sea of data”

Macro-eventHigh-valueActionableIn-context

EDGEAnalytics

StreamAnalytics

FOG

• High Volume• Continuous Streaming• Extreme Low Latency• Disparate Sources• Temporal Processing• Pattern Matching• Machine Learning

Oracle Stream Analytics: From Noise to Value

• HighVolume• ContinuousStreaming• Sub-MillisecondLatency• DisparateSources• Time-WindowProcessing• PatternMatching

• HighAvailability/Scalability• CoherenceIntegration• Geospatial,Geofencing• BigDataIntegration

• BusinessEventVisualization

• Action!

Oracle Stream Analytics - Simplifying Stream Processing18

Page 18: Oracle Stream Analytics - Simplifying Stream Processing

Oracle Stream Analytics Platform

What it does• Compelling, friendly and visually stunning real time

streaming analytics user experience for Business users to dynamically create and implement Instant Insight solutions

Key Features• Analyze simulated or live data feeds to determine event

patterns, correlation, aggregation & filtering• Pattern library for industry specific solutions• Streams, References, Maps & Explorations

Benefits• Accelerated delivery time• Hides all challenges & complexities of underlying real-time

event-driven infrastructure

Oracle Stream Analytics - Simplifying Stream Processing19

Page 19: Oracle Stream Analytics - Simplifying Stream Processing

Oracle Stream Analytics – Self-Service Stream Processing!

Understanding of CQL Filtering, Correlation, Pattern: NOT NEEDED

Understanding of IT Deployment and Management: NOT NEEDED

Understanding of Development, Java, Best Practices: NOT NEEDED

Understanding of the Event Driven Platform: NOT NEEDED

Oracle Stream Analytics - Simplifying Stream Processing20

Page 20: Oracle Stream Analytics - Simplifying Stream Processing

Oracle Stream Analytics – Terminology

Explorer: The Application User Interface Catalog: The repository for browsing resources

Oracle Stream Analytics - Simplifying Stream Processing21

Page 21: Oracle Stream Analytics - Simplifying Stream Processing

Oracle Stream Analytics – Terminology

Stream: incoming flow of events that you want to analyze (CSV, Kafka, JMS, Rest, MQTT, …)

Exploration: application that correlates events from streams and data sources, using filters, groupings, summaries, ranges, and more

Oracle Stream Analytics - Simplifying Stream Processing22

Page 22: Oracle Stream Analytics - Simplifying Stream Processing

Oracle Stream Analytics – Terminology

Shape: A blueprint of an event in a stream or data in a data source. How the business data is represented in the selected stream

Map: collection of geo-fences

Reference: A connection to static data that is joined to a stream to enrich it and/or to be used in business logic and output

Oracle Stream Analytics - Simplifying Stream Processing23

Page 23: Oracle Stream Analytics - Simplifying Stream Processing

Oracle Stream Analytics – Terminology

Pattern: A pre-built Exploration that addresses a particular business scenario in a focused and simplified User Interface

Connection: collection of metadata required to connect to an external system

Targets: defines an interface with a downstream system

Oracle Stream Analytics - Simplifying Stream Processing24

Page 24: Oracle Stream Analytics - Simplifying Stream Processing

Business accessibility to Geo-Streaming Analytics

Real Time Streaming Solutions face an increasing need to track "assets of interest" and initiate actions based on encroachment of boundary proximity to fixed and moving objects and other geographic, temporal, or event conditions.

Geo-Fence,Fence,Polygon

Geo-StreamingOracle Stream Analytics - Simplifying Stream Processing25

Page 25: Oracle Stream Analytics - Simplifying Stream Processing

“Addvaluetoyourrealtimestreamingdatadiscoveryandanalyticsbyapplyingandincludingmathematical,statisticalanalysistotheliveoutputstream”

“Thesestreaming“Excelspreadsheets”reallydocometolife”

Expression Builder enabling calculations

Oracle Stream Analytics - Simplifying Stream Processing26

Page 26: Oracle Stream Analytics - Simplifying Stream Processing

Concept of Connections and their reuse in Streams

Oracle Stream Analytics - Simplifying Stream Processing27

Page 27: Oracle Stream Analytics - Simplifying Stream Processing

Decision Table for Nested IF-THEN-ELSE Rules

Oracle Stream Analytics - Simplifying Stream Processing28

Page 28: Oracle Stream Analytics - Simplifying Stream Processing

Topology View and Navigation

Oracle Stream Analytics - Simplifying Stream Processing29

Page 29: Oracle Stream Analytics - Simplifying Stream Processing

Relationship between Streams (Sources), References and Explorations

Oracle Stream Analytics - Simplifying Stream Processing30

Page 30: Oracle Stream Analytics - Simplifying Stream Processing

Demo

Oracle Stream Analytics - Simplifying Stream Processing31

Page 31: Oracle Stream Analytics - Simplifying Stream Processing

Oracle Stream Analytics Demo Use Case: Truck Movements

Truck DataIngestion Geo-Fencing

2016-06-0214:39:56.605|98|27|MarkLochbihler|803014426|WichitatoLittle RockRoute 2|Normal|38.65|-90.21|5187297736652502631

{"timestamp":"2016-06-0214:39:56.991","truckId":99,"driverId":31,"driverName":"RommelGarcia","routeId":1565885487,"routeName":"SpringfieldtoKCViaHanibal","eventType":"Normal","latitude":37.16,"longitude":"-94.46","correlationId":5187297736652502631}

RecklessDrivingDetector

NEAR

ENTER

TruckDriver

DashboardMovement MovementJSON

RecklessDriver

Oracle Stream Analytics - Simplifying Stream Processing32

Page 32: Oracle Stream Analytics - Simplifying Stream Processing

Continuous Ingestion in Stream Processing

DBSource

BigData

Log

StreamProcessing

IoT Sensor

EventHub

Topic

Topic

REST

Topic

IoT GW

CDCGW

Conn

ectCDC

DBSource

Log CDC Native

IoT Sensor

IoT Sensor

33

DataflowGW

Topic

Topic

Queue

MQTTGW

Topic

DataflowGW

Dataflow

TopicRE

ST33FileSourceLog

Log

Log

Social

Native

Oracle Stream Analytics - Simplifying Stream Processing33

Topic

Topic

Page 33: Oracle Stream Analytics - Simplifying Stream Processing

Apache Kafka – High-volume messaging system

Distributed publish-subscribe messaging system

Designed for processing of high-volume, real time activity stream data (logs, metrics collections, social media streams, …)

Topic Semantic

does not implement JMS standard!

Initially developed at LinkedIn, now part of Apache

Kafka Cluster

Consumer Consumer Consumer

Producer Producer Producer

Oracle Stream Analytics - Simplifying Stream Processing34

Page 34: Oracle Stream Analytics - Simplifying Stream Processing

Demo: Oracle Stream Analytics

Oracle Stream Analytics - Simplifying Stream Processing35

Page 35: Oracle Stream Analytics - Simplifying Stream Processing

Demo: Oracle Stream Analytics

Oracle Stream Analytics - Simplifying Stream Processing36

Page 36: Oracle Stream Analytics - Simplifying Stream Processing

Demo: Oracle Stream Analytics

Oracle Stream Analytics - Simplifying Stream Processing37

Page 37: Oracle Stream Analytics - Simplifying Stream Processing

Demo: Oracle Stream Analytics

Oracle Stream Analytics - Simplifying Stream Processing38

Page 38: Oracle Stream Analytics - Simplifying Stream Processing

Summary

Oracle Stream Analytics - Simplifying Stream Processing39

Page 39: Oracle Stream Analytics - Simplifying Stream Processing

Native Stream Processing => OEP server

Ingestion

Event Source

Event Source

Event Source

Oracle Stream Analytics - Simplifying Stream Processing40

IndividualEvent

PPPPPPPPPPPP

Page 40: Oracle Stream Analytics - Simplifying Stream Processing

Micro-Batch Stream Processing => Spark Streaming

Ingestion

Event Source

Event Source

Event Source

Oracle Stream Analytics - Simplifying Stream Processing41

PPPPPP

Page 41: Oracle Stream Analytics - Simplifying Stream Processing

Summary

Oracle Stream Analytics leverages the capabilities found in Oracle Event Processing (OEP)

Empowering Business users to gain insight into real-time information and take appropriate actions when needed => makes stream processing accessible

Makes Stream/Event Processing less technical => “Excel spread sheet” on Streams

Part of Oracle IoT Cloud Service

Support Spark Streaming as a deployment platform for Streaming ML

Interesting road map: Rule Engine, Machine Learning, Extensible Patterns

Oracle Stream Analytics - Simplifying Stream Processing42

Page 42: Oracle Stream Analytics - Simplifying Stream Processing

Oracle Stream Analytics on Docker

Oracle Stream Analytics 12.2.1 Documentation

Oracle Stream Analytics 12.2.1 Download

Oracle Stream Analytics - Simplifying Stream Processing44

Page 43: Oracle Stream Analytics - Simplifying Stream Processing

Guido SchmutzTechnology Manager

[email protected]

Oracle Stream Analytics - Simplifying Stream Processing45