safe harbor statement - wordpress.com...2017/09/04 · hive tables –query oracle nosql data from...

Copyright © 2017, Oracle and/or its affiliates. All rights reserved. |

Safe Harbor Statement

The following is intended to outline our general product direction. It is intended for information purposes only, and may not be incorporated into any contract. It is not a commitment to deliver any material, code, or functionality, and should not be relied upon in making purchasing decisions. The development, release, and timing of any features or functionality described for Oracle’s products remains at the sole discretion of Oracle.

Confidential – Oracle Internal/Restricted/Highly Restricted 2


Enterprises use NoSQL because it


Enables elastic scale to support dynamic workloads

Supports flexible data models to support any type of data

Delivers extreme read and write capabilities for demanding applications

Simplifies data management for any type of applications

Lowers data management cost.


• Developer Focus

– BASE & ACID xtions

– Tables / JSON / Binary

– C, C#, Java, Python & Node.js APIs

– Secondary Indexes

• IT Operations Focus

– Elastic Configuration

– Secure Access

– Data Center Support

– Online management

• Differentiating Features

– ACID transactions

– Rich SQL dialect

– Online rolling upgrades

– Streaming large object support

– Strong Oracle technology Integration

– Engineered Systems and Commodity HW

6

Oracle NoSQL Database - Enterprise Ready

Best NOSQL choice for any company using Oracle products


Basic Design


Advanced Key-Value Database


Key-Value

Large Object Avro Tables

JSON ElasticSearch Streams Hive


capacity=x capacity=y

capacity=…

capacity=…

capacity=…

Storage Nodes and Zones


Zone 1 Zone 2


Replication Nodes and Shards


Zone 1 Zone 2

rn rn

rn

rn rn

rn

rn rn

rn rn

M R

R

M R

R

R R

R R

Shard 1

Shard 2

ZRF=3 ZRF=2


Writing


Zone 1 Zone 2

M R

R

M R

R

R R

R R

Shard 1

Shard 2


Reading


Zone 1 Zone 2

M R

R

M R

R

R R

R R

Shard 1

Shard 2


Scalability


Shard 1

Shard 2

Oracle NoSQL Client

partition P1 partition P2 ….

….partition Pn+1partition Pn

key

hash(key)

partition #

LUT


Configurable CAP

• Greater Flexibility

– Configurable • Durability per operation

• Consistency per operation

– ACID by default

– Transaction scope is single API call

– Records share same shard key

– Multiple operations supported

14


Oracle NoSQL Database – Key value example

// Construct the keyArrayList<String> majorK = new ArrayList<String>();ArrayList<String> minorK = new ArrayList<String>();

majorK.add(“Item1”);minorK.add(“Property1”);

Key myKey = Key.createKey(majorK, minorK);

// The Data portion can be any serializable chunk of information. Here we will use a simple Java string.String myData = “This is some random text that represents the note’s content.”Value myValue = Value.createValue(myData.getBytes());

// Write the Key/Value pair to the store:kvstore.put(myKey, myValue);

What is a Key-Value ?

Key Value

Major key componentMandatory, all records sharing the same major key are allocated to the same shard

Minor key componentOptional; the key is the concatenation of the major key and the minor key components

Key Value

Copyright © 2017, Oracle and/or its affiliates. All rights reserved. | Confidential – Oracle Internal/Restricted/Highly Restricted 16

Tables

UserId PolicyId lname fname email

[Value]Primary Key

Shard Key

CREATE TABLE myUsers (UserId INTEGER,PolicyId INTEGER,lname STRING,fname STRING,email STRING,PRIMARY KEY (SHARD(UserId), PolicyId)

)

Complex types supported: JSON and maps


Oracle NoSQL Database – SQL Query

17

Driver

Application

SELECT-FROM-WHERE,ORDER BY, OFFSET, LIMIT,UPDATE+SQL-like DDLCREATE, ALTER and DROP TABLE and INDEX

SQL Commands Now Available in NoSQL

Declarative query language for developers more comfortable with SQL than API level access.


Oracle NoSQL Database – UPDATE statement for SQL



Large Object Support

• Parallel Streaming Interfaces

– Efficient storage and retrieval of large objects

– Client side streaming interface for low memory consumption

– Server side splitting and distribution of object chunks across nodes for better read/write latency

– Automatic partial LOB detection

19

Large

Object

No

SQL

DB

Dri

ver

Ap

plic

atio

n

Shard 2

Shard N

Shard 1


Oracle NoSQL Database – Bulk Put API

20

Serial Data Store Bulk Put API

Data 1 API Call

Data 2

Data 3

Data 4

Data 1

API Call

Data 2

Data 3

Data 4

Total = 4 API Calls Total = 1 API Call

API Call

API Call

API Call

Database

Database


Oracle NoSQL Database – Full Text Search (FTS)

21

ElasticSearch

3 - Oracle NoSQL DB Bulk-GETof primary keys

4 - Oracle NoSQL DB returnsthe records

ElasticSearch Populated Asynchronously

ElasticSearch

Oracle NoSQL DB – write data

1 – Search ES for content

2 – Returns list of primary keys of matching records

Write to the database Read from the database


Oracle NoSQL Database – Kerberos Integration

22

NoSQL Database

Authentication Service

Ticket Granting Service

Kerberos Distribution Center

I am “[email protected]” , Please prove this to NoSQL DB

Here is a Ticket That Proves you are [email protected]

Here is a Kerberos Ticket

for [email protected]

1

3

2

mailto:[email protected]

mailto:[email protected]


Oracle NoSQL Database – Time-To-Live (TTL)• Automatic expiration of persistent records.

• An application user can specify a TTL duration for each record.

• The record would expire logically after the duration. The user can also update the TTL before a record expires.

23

Record 1

Expires in 2 hours

Record 2

Expires in 3 hours

Record 3

Expires in 24 hours

Record 4

Expires in 30 hours

Record 5

Never expire

Record 1

Record 2

Record 3

Record 4

Record 5

Record 2

Record 3

Record 4

Record 5

Record 3

Record 4

Record 5

Record 4

Record 5 Record 5

Time – Not To Scale

Dat

abas

e S

ize


Oracle NoSQL Database Streams API

• Changes to the Store can be asynchronously forwarded to stream processing application

• Compliant with http://www.reactive-streams.org



Drivers


Driver

Application JavaPythonNode.jsRESTCC#


NoSQL DB integration - Hadoop

HDFS

HDFS Files

REDUCE

MAP

MAP

MAP

MAP

MAP

MAP

REDUCE

REDUCE


NoSQL DB integration – Oracle RDBMS

Externaltable

SELECT name FROM unstruct_people

SQL engine

nosql_streampreprocessor


Integrating the best of breed

Oracle Big Data SQL

One fast SQL query, on all your data, in parallel


Integration with Apache Hive and Oracle Big Data SQL

• Oracle Big Data SQL

– Common SQL access layer to data stored in Hadoop HDFS, Hive and Oracle NoSQL Database

– Defines new external table access driver types. ORACLE_HIVE for accessing data stored in or referenced by Hive Tables

– Query Oracle NoSQL Data from Hive or Oracle Database

– Runs Map Reduce Jobs against data stored in Oracle NoSQL Database

• Support for primitive and complex data types

• Common data definition between Oracle NoSQL DB, Hive and Oracle Database

29

BIG DATA SQL


NoSQL DB integration – Oracle Coherence

NoSQL DB Driver

TB

Mem

ory


Oracle Big Data Spatial and Graph (BDSG)

Property Graph Database

Spatial Analysis Features

31


Oracle Big Data Spatial and Graph

Access Layer

Oracle Big Data Spatial and GraphProperty Graph Architecture

Graph Analytics

Apache Blueprints & Lucene/SolrCloud RDF (RDF/XML, N-Triples, N-Quads,

TriG,N3,JSON)

REST/W

eb

Service

Java, Gro

ovy, P

ytho

n, …

Java APIs

Java APIs/JDBC/SQL/PLSQL

Property graph formats supported

GraphMLGML

Graph-SON Flat Files

CSVRelational Data Sources

32

Oracle NoSQL Database

Apache HBase

Parallel In-Memory Graph Analytics (PGX)


Where is NoSQL used?

33

Simple Data Management

Globally Distributed,

“Always On” data

Competitive Advantages of

“Fast Data”

Lower TCO, commodity

HW scale-out

ERP

EAM

Inventory Control

Accting & Payroll

Process Mgmt

Business Analytics

CRM

…

Driver

ApplicationReal Time Event

Processing

Customer Portals

Online Banking

…

Mobile Data Management

Time Series & Sensor Data Mgmt


Choose the RIGHT storage option for the job

Hadoop Distributed File System (HDFS)

Oracle NoSQL Database Oracle Database

File System Key-Value Database Relational Database

No inherent structure Simple data structure Complex data structures, rich SQL

High volume writesHigh volume random reads and

writesHigh volume OLTP with 2-PC

Limited functionality,roll-your-own applications

Simple get/put high speed storage, flex configuration

Security, Backup/Restore, Data life cycle mgmt, XML, etc.

Batch OrientedReal-Time, web-scale specialized

applicationsGeneral purpose SQL platform, multiple

applications, ODBC, JDBC

34


Oracle NoSQL Database - Flavors

Enterprise Edition (EE) Basic Edition (BE) Community Edition (CE)

Connectors to Oracle Products Same as CE Open Source

Enhanced Security Same as CE Full Performance, Same API

Licensed per OCPU License with BDEE List Price - $ 0

Support Available ($) Support available with DB EE support

Support Available ($) / Community

Commercial Oracle License Licensed together with Oracle DB EE

AGPL V3 license


Common NoSQL applications

• Customer portals (browser, mobile…)

• Real-time event processing

• Mobile Data Management

• Time-series data management

• Online banking

36

• Customer 360

• Customer loyalty programs

• Profile management

• Product catalog

• Content management

• Digital communication (e.g. “click-to-chat”)

• …


Finance

• Large credit card network provider

• Fraud rule and history management

• 6,500 ops/sec, 85% update on read

Manufacturing

• Large airline manufacturer

• Developing next generation planes

• 750K key inserts/sec, 6TB/hour

37

Telecoms

• Large mobile telecom operator

• Mobile personalized advertizing

• 24,000 ops/sec, 100% update on read

Customer Loyalty

• Operationalize analytics to NoSQL Customer Portal

• Addressed processing conflicts

• 20M+ customers in multiple countries

Oracle NoSQL Database customer success…

……many more

Airbus


Benefits

Scalable multi-channel customer portal

Predictable performance for all operations

Reduced time to market

Easy application evolution

Coupon redemption, Vendor recommendation

NoSQL for Customer Loyalty

Objectives

Scalable customer loyalty portal

New multi-channel consumer model

Improve operational efficiency

Solution

Personalized multi-channel coupon generation and redemption

Cross-promote affiliated vendors

Scale system with customers and participating retailers

NoSQL DB DriverApplication

Retail Partners

Customer Profiles

End Customers

Available CouponsMarket Segmentation

38

http://www.payback.net/


Financial Services coordinated theft prevention

NoSQL for Fraud Scoring

Objectives

Solution

Benefits

Application Data Ingestion

Tran

sact

ion

Au

tho

riza

tio

nP

roce

sso

r

Combine data sources for complex scoring

Detect, alert analyst with low latency

Handle burst seasonal transaction volumes

Oracle Coherence cluster for real time transaction object management

Oracle NoSQL Database for fraud model and customer profile management

Oracle Database for statistics and fraud modeling-related data

Simple data model, ACID transactions

Scalability, Reliability, Low Latency

Elasticity of sharded data repositories

Easy configuration and administration

NoSQL DB Driver

39


Objectives

Solution

Benefits

• Increase scalability of data storage•Deliver higher concurrency analytic data access•Scale data loading independently from analysis•Commercial support for mission critical system

•Oracle NoSQL database for high speed storage and range based extraction of time series data.•Oracle NoSQL database for agile schema, replaced

HDF5 storage format, kept analysis client program•Oracle Big Data Appliance for efficient manageability

and lowest TCO•Hadoop post processing and RDBMS connectivity to

Enterprise systems

• Improve scale of storage for flight test sensor data• Increase concurrency of access to data for analysis• Improve system availability for analysts by allowing

simultaneous data ingestion and analysis

Big Data Appliance

NoSQL DB Driver

Event Ingestion and Extraction

NoSQLDB/Oracle RDBMS

Hadoop/Oracle RDBMS

Oracle or Any third parties

SQL/Data Analytics Tools

NoSQL for Sensor Event Storage & ProcessingLarge scale sensor data capture and analysis

40

Airbus


Product Recommendation Platform

41

Elastic Data Store

RecommendationRule Engine

SourceSystems

Admin

Customers

Corporate Web Page Recommendation Platform

Rules

CustomerInfo

Recommend

User Info, Billing Info,etc

Request RespondLogin

User ID


Performance Comparison

42

0

10

20

30

40

50

60

70

0 2000 4000 6000 8000 10000 12000 14000 16000 18000 20000

# o

f se

rve

rs

Transactionsper second

NoSQL HDD

NoSQL SSD

HBASE HDD

1. Oracle NoSQL Database with SSDs achieves 26000 tps with 3 nodes.2. Compared to HBase, Oracle NoSQL Database achieves high performance goals with much smaller number

of nodes.3. Considering high performance goals combined with additional future requirements, the total cost of Oracle

NoSQL Database is estimated to be much lower than HBase.

Over 20,000 tpscan be handled

with just 3 nodes


Oracle NoSQL DB competition

43

Oracle NoSQL Database

Linear Scalability 200+

nodes

ACID, Configurable

transaction semantics

Flexible Schema – KV,

JSON, Table

Only NoSQL store with

Enterprise level support.

Easier to setup,

upgrades

Better performance &

predictability.

Tight integration with

Oracle ecosystem –

Database, FMW, Big

Data Appliance

Vs. MongoDB

Well known scalability

problems. (does not

scale well beyond 6

nodes)

Lacks write concurrency

due to collection level

locking. Can lose data.

Update and delete

operations perform

poorly.

Schema Bloat

3-6X slower than Oracle

NoSQL DB.

Vs. Cassandra (Datastax)

Inconsistent reads due

to background

operations.

Columnar, KV, Table

Poor performance when

used with Hadoop.

Difficult to keep data in

sync across the cluster.

Hard to manage large

clusters.

Inefficient I/O and

memory usage.

2-3X slower than Oracle

NoSQL DB.

Vs. HBase

Clusters are complex to

run and hard to manage.

Columnar, KV, Table

Less flexible transaction

model. Can lose updates.

High memory usage.

Inconsistent performance

Difficult to configure.

Less flexible transaction

model.

2-3X slower on inserts

Performance, reliability, scalability, integration

Not Integrated with the Oracle Stack


Performance comparisonOracle NoSQL DB, Cassandra, Mongo DB

Benchmark specifications

• Yahoo! Cloud Serving

Benchmark (YCSB)

[13 byte key; 1100 byte value]

• 60 Million records in database

(~60 GB of data)

• 3 Node system

(X4170 m2: 2 Xeon(R) X5670

2.93GHz; 72 GB; SSD drives)


• Much better READ Performance• Lower READ Latency

• Stable Performance

• Linear Scalability

Setup

• Replication Factor (RF) = 3

• Each BDA node has 12 Disk

NoSQL DB Highlights

HBase vs Oracle NoSQL DatabaseYCSB Benchmark Results

0

10

20

30

40

50

60

0

10,000

20,000

30,000

40,000

50,000

60,000

70,000

6 (672M) 12 (1344M)

Ave

rage

Lat

en

cy (

ms)

Thro

ugh

pu

t (o

ps/

sec)

BDA Nodes (record count)

95/5 Read/Update Throughput

NoSQL Throughput (ops/sec) HBase Throughput (ops/sec)

NoSQL Read Latency (ms) Hbase Read Latency (ms)


Predictability, Reliability & Support

• Global, mission-critical application deployment experience

• Decades of enterprise-grade non-relational database technology

• Oracle Support available for both Enterprise and Community Edition

• Designed for Predictability and Manageability

Bulk Insert Test Cluster Expansion Test Rolling Upgrade Test

46


YCSB on SSD-backed commodity servers

• What’s the big deal

– Twitter sees ~500M tweets/day• This is 350K a minute

• Capture all tweets with 3 commodity servers

– 1.25M ops/sec

– 2 billion records

– 2 TB of data

– 95% read, 5% update

– Low latency, High Scalability

47

0

1

2

3

4

0

200,000

400,000

600,000

800,000

1,000,000

1,200,000

1,400,000

6 (2x3) 12 (4x3) 24 (8x3) 30 (10x3)

Ave

rage

Lat

en

cy (

ms)

Thro

ugh

pu

t (o

ps/

sec)

Cluster Size ( 3->15 servers )

Mixed Throughput

Throughput (ops/sec) Write Latency (ms)

Read Latency (ms)


Online Rolling Upgrades

• Linear time impact for upgrade– Admin commands available to describe

safe upgrade order– Scripted available hands-free upgrade

experience– Read/Write availability throughout the

upgrade process

200 Node version upgrade in 15 minutes

48

0

2.5

5

7.5

10

12.5

15

17.5

72 (24x3) 144 (48x3) 216 (72x3)

Tim

e t

o U

pgr

ade

(m

in)

Total Nodes (Shards x Rep. Factor)

Online Rolling Upgrade


Cluster expansion while running YCSB

• What we’re showing

– YCSB running on a 144-node system

– Add 50% more HW

– Gradual increase in throughput, decrease in latency

– Ability to increase workload by 50%

– No interruption to YCSB application

49

0

1

2

3

4

5

6

7

8

0

10,000

20,000

30,000

40,000

50,000

60,000

70,000

80,000

144(48x3)

20% 40% 60% 80% 216(72x3)

216*(72x3)

Ave

rage

Lat

en

cy (

ms)

Thro

ugh

pu

t (o

ps/

sec)

Nodes (Shards x RF)

95/5 Read/Update Throughput

Throughput (ops/sec) Read Latency (ms)

Update Latency (ms)

* = Clients increased as well

safe harbor statement - wordpress.com...2017/09/04 · hive tables –query oracle nosql data from...

Documents