processing twitter events in real-time with oracle event processing (oep) 12c

2013 © Trivadis

BASEL BERN BRUGG LAUSANNE ZUERICH DUESSELDORF FRANKFURT A.M. FREIBURG I.BR. HAMBURG MUNICH STUTTGART VIENNA

2013 © Trivadis

Echtzeitverarbeitung von Twitter Streams mit Oracle Event Processing (OEP) 12c

Guido Schmutz

DOAG Konferenz 2014

19.11.2014 – 15:00 Raum Kiev

19.11.2014 Processing Twitter Stream with Oracle Event Processing (OEP)

1

INFOBOX – Read and delete •  A heading and an optional sub-heading

can be placed on the first slide. •  The title is written directly under the

name (Shift+Return) •  If multiple speakers are named, please

just write the names one underneath the other (there is normally no space for titles, etc.)

2013 © Trivadis

Guido Schmutz

•  Working for Trivadis for more than 17 years

•  Oracle ACE Director for Fusion Middleware and SOA •  Co-Author of different books •  Consultant, Trainer Software Architect for Java, Oracle, SOA and

Big Data / Fast Data •  Member of Trivadis Architecture Board •  Technology Manager @ Trivadis

•  More than 25 years of software development experience

•  Contact: [email protected] •  Blog: http://guidoschmutz.wordpress.com •  Twitter: gschmutz


2

2013 © Trivadis

AGENDA

1.  Introduction

2.  Twitter Use Case

3.  Processing with Oracle Event Processing (OEP)

4.  Visualization with Oracle Business Activity Monitoring (BAM)

5.  Event Processing in Information Architecture


3

INFOBOX – Read and delete •  If the agenda is used as an interim

page, please highlight the relevant chapter in red font.

•  To allow optimum alignment of objects,

display the drawing guides (right-click on the page)

2013 © Trivadis

Big Data Definition (4 Vs)


4

+ Time to action ? – Big Data + Event Processing = Fast Data

Characteristics of Big Data: Its Volume, Velocity and Variety in combination

2013 © Trivadis

The world is changing …

The model of Generating/Consuming Data has changed ….

Old Model: few companies are generating data, all others are consuming data

New Model: all of use are generating data, and all of us are consuming data


5

2013 © Trivadis


6

2013 © Trivadis

Internet Of Things – Sensors are/will be everywhere

There are more devices tapping into the internet than people on earth

How do we prepare our systems/architecture for the future?


7 Source: Cisco Source: The Economist

2013 © Trivadis

AGENDA

1.  Introduction






8

2013 © Trivadis

Retrieve Tweets and Visualize


9

2013 © Trivadis

Access to Tweets


10

Quelle

Source Limitations Cost Twitter’s Search API 3200 / user

5000 / keyword 180 requests / 15 minutes

free

Twitter’s Streaming API 1%-40% of total volume free

DataSift none 0.15 -0.20$ /

unit Gnip none On request

2013 © Trivadis

AGENDA

1.  Introduction






11

2013 © Trivadis

Oracle Event Processing (OEP) - Engine

Lightweight Java Application Server

•  Full environment for running Java applications

•  Module Framework - OSGi

High Throughput

•  Hundreds of thousands of events/second

Event Processing Infrastructure

Easy-to-use development environment

•  Service Framework – Spring DM, POJO

Multiple-choice VM

•  JRockit or WebLogic RealTime


12

2013 © Trivadis

Oracle Event Processing Application

Adapter •  Provide a way for EPN to exchange data with components that are external to network •  Either an Event Source (inbound) or an Event Sink (outbound)

Processor •  Contain query code written in CQL and consumes event data from a preceding

channel •  May also generate new events to an output channel •  CQL is an extension of SQL


13

2013 © Trivadis

Oracle Event Processing Application

Bean Bean

•  Provide a place for logic written in Java •  Java code can receive events (Event Sink) and/or send events (Event Source)

•  Are implemented using the Spring Framework model or the OEP event bean model Channel

•  Two types of channel exist: Stream and Relation

CQL Processor •  Executes Continous Query Language (CQL) queries •  Used to declaratively describe logic (queries) to be applied on the event stream


14

2013 © Trivadis

Oracle Event Processing - CQL

CQL is an extension to Structured Query Language (SQL) •  Same keywords and syntax rules, but additional support for streaming data

event conceptually corresponds to a row in a database table, but •  one event is always before or after another in time

•  stream is potentially infinite and ever-changing


15

Data Query

Result Tuples Result Tuples

Traditional Database CEP Engine

QueryData

2013 © Trivadis

Oracle Event Processing – Visual Development Tools


16

2013 © Trivadis

Oracle Event Processing – Operation & Management


17

2013 © Trivadis

Implementation – complete picture


18

Mention Extractor

Twitter Adapter

CounterProcessor

Hashtag Extractor

Author Extractor

BAM Tweet

BAM Counter

Working on my OEP & BAM demo for the session

”Echtzeitverarbeitung von Twitter Feeds …” #doag2014

#oep12c #bam12c http://bit.ly/1xafUZJ

gschmutz

#doag2014 #oep12

#bam12c,2

gschmutz,1

@doag2014,5 #oep12c,1 JMS

JMS Twitter

range 30 seconds slide 30 seconds

Sentiment Enricher

Working on my OEP & BAM demo for the session ”Echtzeitverarbeitung von Twitter Feeds …” #doag2014 #oep12c

#bam12c http://bit.ly/1xafUZJ

1

#bam12c

2013 © Trivadis

1) Creating a Twitter Adapter


19

Twitter Adapter




Twitter

2013 © Trivadis

2) Send Tweets to BAM


20

Twitter Adapter

BAM Tweet




JMS Twitter Sentiment Enricher



1

2013 © Trivadis

3) Extract interesting information from Tweet


21

Mention Extractor

Twitter Adapter

Hashtag Extractor

Author Extractor

BAM Tweet




gschmutz

#doag2014 #oep12

JMS Twitter Sentiment Enricher



1

#bam12c

2013 © Trivadis

4) Count occurrences within period


22

Mention Extractor

Twitter Adapter

CounterProcessor

Hashtag Extractor

Author Extractor

BAM Tweet

BAM Counter




gschmutz

#doag2014 #oep12

#bam12c,2

gschmutz,1

@doag2014,5 #oep12c,1 JMS

JMS Twitter


Sentiment Enricher



1

#bam12c

2013 © Trivadis

Implementing in Oracle Event Processing


23

Mention Extractor

Twitter Adapter

CounterProcessor

Hashtag Extractor

Author Extractor

BAM Tweet

BAM Counter

JMS

JMS Twitter


Sentiment Enricher

2013 © Trivadis

1) Creating Twitter Adapter – Connecting to Twitter Stream


24

2013 © Trivadis

1) Creating Twitter Adapter – Tweet Event


25

2013 © Trivadis

1) Creating Twitter Adapter – Adapter Factory


26

2013 © Trivadis

1) Creating Twitter Adapter – Assembly


27

2013 © Trivadis

1) Creating Twitter Adapter – Export Adapter to server


28

2013 © Trivadis

1) Creating Twitter Adapter – Using Twitter Adapter


29

2013 © Trivadis

2) Sentiment Enrichment


30

2013 © Trivadis

3) Sending Tweets to BAM

Using Oracle BAM Enterprise Message Sources (JMS) interface


31

2013 © Trivadis


32

3) Sending Tweets to BAM – Convert event to JMS MapMessage

2013 © Trivadis

Oracle BAM: Architected for Integration and Visualization

Processing Twitter Stream with Oracle Event Processing (OEP) 19.11.2014

38

2013 © Trivadis

Architectural Pattern: Standalone Event Stream Processing


43 43

Event Processing (ESP / CEP)

State Store / Event Store En

terp

rise

Even

t Bus

(In

gres

s)

Even

t Cl

oud

Internet of Things

Social Media Streams

Ente

rpris

e Ev

ent B

us

43

Analytical Applications

DB

Ente

rpris

e Se

rvic

e Bu

s

Business Rule Management

System Rules

Event Processing

Result Store

2013 © Trivadis

Hadoop Big Data Infrastructure

Architectural Pattern: Event Stream Processing as part of Lambda Architecture


44 44


State Store / Event Store

Ente

rpris

e Ev

ent B

us

(Ingr

ess)

Even

t Cl

oud

Internet of Things


Ente

rpris

e Ev

ent B

us

44


DB

Ente

rpris

e Se

rvic

e Bu

s

Event Processing

Map/Reduce HDFS Result

Store

Result Store

2013 © Trivadis

Hadoop Big Data Infrastructure

Architectural Pattern: Event Stream Processing as part of “Kappa” Architecture


45 45



Ente

rpris

e Ev

ent B

us

(Ingr

ess)

Even

t Cl

oud

Internet of Things


45


DB Ente

rpris

e Se

rvic

e Bu

s

Event Processing

Replay HDFS

Result Store

2013 © Trivadis

Even

t Cl

oud

Event Processing in modern architecture


46

Enterprise Applications

WS

REST

JMS

RDMBS

Loca

l ESB

External Cloud Service Providers

Ente

rpris

e Se

rvic

e Bu

s (E

SB)

EJB



BPM and SOA Platform

Event

REST

Business Logic/Rules

NoSQL


Data Analytics

Internet of Things

Event Processing

46

Mobile Apps

DB

Rich (Web) Client Apps

DB


Ente

rpris

e Ev

ent B

us

(Ingr

ess)

Ente

rpris

e Ev

ent B

us

Visualization

Biz Logic Rules

WS

Event

Business RuleManagement

System Rules

Processes ACM

HumanWF

Oracle Event Processing for Java Embedded

Coherence

Oracle NoSQL Business Activity Monitoring (BAM)

Oracle RDBMS Se

rvic

e Bu

s

SOA Suite

BPM Suite

Web

Logi

c JM

S

Web

Logi

c JM

S ED

N

Oracle Rules

Oracle Event Processing (OEP) Oracle NoSQL Oracle RDBMS

Business Activity Monitoring (BAM)

API G

atew

ay

EDN

Oracle Real-Time Decisions (RTD)

2013 © Trivadis

Questions and answers ...

2013 © Trivadis

BASEL BERN BRUGG LAUSANNE ZUERICH DUESSELDORF FRANKFURT A.M. FREIBURG I.BR. HAMBURG MUNICH STUTTGART VIENNA

Guido Schmutz

Technology Manager

[email protected]


INFOBOX – Read and delete •  There are two versions of the last slide

available, one for the contact details of a speaker, and one for two or more speakers.

•  Name, title and location always underneath one another in one row (Shift+Return)

•  This idea is that this is the last slide (also for questions and answers) and is on the screen for a long time at the end of the presentation, so the viewers have the chance to write down the contact data J

47