Real Time, In-Process Analytics at Substantially Lower Costs with IBM and Redis LabsLeena JoshiVP Product Marketing

2

Agenda

• The business driver for real time, in-process analytics

• Why Redis for transactional & analytic scenarios

• IBM Power + Redis Labs : Horsepower at substantially lower costs

• Next Steps

The Business Driver

4

The new standard for E2E application response time, under any load100msec

Average roundtrip internet latency50msec

Required roundtrip app response time (includes processing & multi-DB access)

50msec

Required DBresponse time

1msec

Database

App Servers

The Speed of Business

5

Why Do You Need Analytics At the Same Speeds?5

Lost Revenue Competitive RiskBetter User Experience

An offer made in context is more powerful than trying to

reach the customer later

Anticipating intelligently what the customer is likely to do

next makes for a fantastic user experience

A competitor offering a better experience or better offer will

grab market and revenue share

“We think you will also like..” “Let us do this for you...” “We do better than..”

Why Use Redis in Analytics

7

Who We Are

The open source home and commercial provider of Redis

Open source. The leading in-memory database platform, supporting any high performance transactional or analytics use case.

8

Redis Tops Database Popularity Rankings

……..#1 NoSQL in User Satisfaction and Market Presence

……..#1 NoSQL among Top 10 Data Stores

……..#1 database on Docker

#1 NoSQL database deployed in containers

………#1 in growth among top 3 NoSQL databases

………#1 database in skill demand

………# 1 database in Top Paying Technologies

9

Redis is a Game Changer

Simplicity(through Data Structures)

Extensibility (through Redis Modules)

Performance

ListsSorted Sets

Hashes Hyperlog-logs

Geospatial Indexes

Bitmaps

SetsStrings

Bit field

10

• Used by developers like “Lego” blocks

• Enables data to be processed on the database level rather than the application level

• Turns complex functionality into a single command such as:"Get the e-mail address of the user with the highest bid in an auction that started on July 24th at 11:00pm PST”ZREVRANGE 07242015_2300 0 0

Simplicity: Data Structures - Redis’ Building Blocks

• Enable solving complex problems by creating relations between data structures, using standard or custom (Lua) commands

• The result: cleaner, more elegant code, faster execution time

ListsSorted Sets

HashesHyperlog-

logs

Geospatial Indexes

Bitmaps

SetsStrings

Bit field

11

Extensibility: Modules Extend Redis Infinitely

• Add-ons using a Redis API for seamlessly adding to it use cases and data structures

• Modules enjoy Redis’ simplicity, super high performance, infinite scalability and high availability

• Modules can be created by anyone. Certified by Redis Labs.

Full Text Search Enhanced JSON Graph Operations Secondary Indexes

Linear Algebra SQL Support Image ProcessingN-Dimension

Queries …

12

Performance: The Most Powerful Database

Highest Throughput at Lowest Latency in High Volume of Writes Scenario

Lowest number of servers needed to deliver 1 Million writes/second

300

50 50

20

50

100

150

200

250

300

350

Benchmarks performed by Avalon Consulting Group Benchmarks published in the Google blog

13

Popular Redis Use Cases

Geo SearchData Ingestion Social Functionality

Following, Followers, Relations Location-based ApplicationsHigh Throughput Buffering

Job & Queue Caching

Any Business Application Any Web or Mobile App

High Speed Transactions Time-Series

Business Applications

Analytics

Real-time Computations Time-Based Analysis

14

Example : Redis For Bid Management

The Application Problem

• Many users bidding on items• Need to instantly show who’s

leading, in what order and by how much

• May also need to display analytics like how many users are bidding in what range

• Disk-based DBMS-es are too slow for real-time, high scale calculations

Why Redis Rocks This

• Sorted sets automatically keep list of users and scores updated and in order (ZADD)

• ZRANGE, ZREVRANGE will get your top users

• ZRANK will get any users rank instantaneously

• ZCOUNT will return a count of users in a range

• ZRANGEBYSCORE will return all the users in a range by their bids

15

Redis Sorted Sets

ZADD item:1 10000 id:2 21000 id: 1ZADD item:1 34000 id:3 35000 id 4ZINCRBY item1:1 10000 id:3

ZREVRANGE item:1 0 0id:3

Item: 1id:3 44000

id:4 35000

id:1

id:2

21000

10000

16

Example : Redis For RecommendationsThe Application Problem

• Users, items, likes, dislikes, similarities• Set comparisons of user likes, user

dislikes should help create similarity scores, which can then be stored in a sorted set

• Set comparisons of similar user likes/dislikes with items not purchased by current user should yield suggestions

• High speed and low latency requirements

Why Redis Rocks This• Redis Sets are unordered collections

of strings- SADD to add objects to each tag

• Set operations executed in –memory, blazing fast speeds

• SINTER, SINTERSTORE to intersect

multiple sets

• SUNIONSTORE to add multiple sets

• SISMEMBER to determine membership,

SMEMBERS to retrieve all values

• Sets and Sorted sets combined are a great choice for recommendation engines

17

Redis Sets

SADD item:1 tag:1 tag:22 tag:24SADD tag:1 item:1SADD tag: 2 item:22 item:14 item:3

SINTER tag1 tag2item:3

SUNIONSTORE tag:x tag1 tag2SMEMBERS tag:xitem:1 item:3 item:22 item:14 item:3

item 1 {tag:1, tag:22, tag:24}

{item:1, item:3}tag 1

{item:22, item:14, item: 3}tag 2

{item:1, item:22, item:14, item: 3}tag x

18

Customers Use Redis for Real Time Transactions & Analytics

LARGE DAILY DEALS COMPANY LARGE ONLINE MOBILE AD COMPANY

Why Redis :

• Extreme throughput at low latencies –essential for simultaneous transaction and analytic processing

• At very high transactional rates, impossible to use a different datastore with equivalent ability

• Ease of implementation – built-in data structures reduce the complexity involved in both simple and complex analytics

• Redis used to store and serve the most up-to-date

offers/coupons to customers (Strings, Hashes, Lists)

• Redis also used for personalized recommendations for

customers purchasing coupons(Sets, Sorted Sets, Strings)

• Redis used as datastore to serve mobile ads

• Redis also used to store which ads are being served how

often, top revenue generators and other drivers of financial

and business reporting

19

Intuit: Redis for Recommendations

• Quickbooks self-employed users get rules that auto-categorize transactions

• Increases the wow factor of the app, 60% of people who use the recommended rules subscribe

• Creating and applying recommended rules is managed through the Redis backend

• Redis handles 1000s of simultaneous channels, does not even blink

20

Scopely: Redis for Probabilistic Analysis• Scopely, next generation mobile entertainment• Needs to generate on-the-fly game insights so

games can be tailored to user preferences by location, demographic etc

• 2.8 million events/min, 2.4 billion events/day• Redis powers their real time system for

operational monitoring/business alerts• Ongoing analysis of current game performance,

user engagement vs past• Hyperloglog for estimation of different things –

examples: cheating likelihood, anomalous installs, game play times

Analytics architecture

21

New Modules for Analytics Use Cases

Powerful text search

redisearch redabloom topk countminsketch

Bloom filter Top k most frequent Counts of observations

…

Redis & Spark

23

Spark Operation w/o Redis

Read to RDD Deserialization Processing Serialization Write to RDD

Analytics & BI

1 2 3 4 5 6

Data SinkData Source

24

Spark SQL &Data Frame

Spark Operation with Redis

Data Source Serving Layer

Analytics & BI

1 2

Processing

Spark-Redis connector

Read filtered/sorted

data

Writefiltered/sorted

data

25

Accelerating Spark Time-Series with Redis

Redis is faster by upto 100 times compared to HDFS and over 45 times compared to Tachyon or Spark

Cost Effective Analytics: IBM Power and Redis Labs

27

Why Redis on Power 8

• 800+ databases on a single P8 box:

‒ Redis is single threaded - each DB runs on a single core

‒ P8 supports up to 8 virtual cores on each physical core, i.e. up to 196

virtual cores in a box

‒ RLEC allows running multiple DBs on a single virtual core with no

performance degradation

• Flash, used as RAM extender, provides significant

deployment cost savings

• Strong networking capability, eliminates

packets/sec and bandwidth bottlenecks and keeps

latencies at <1msec

CAPI(Coherent

Accelerator Processor Interface)IBM Power 8

(24 cores, 192 vcores, RAM, Flash)

…

28

Redis on Flash – A New ConceptFlash used as RAM extender

29

How to Achieve Optimal Price/Performance

By dynamically setting RAM/Flash ratio Behind the scenes…

30

Deployment Options

Standard Servers IBM Data Engine for NoSQL Appliances

S822 LC32GB -1 TB memory

S822L or S812LExternal IBM Flash

ORS822LC + 2 TB FlashS812L + 4 TB Flash S822 L + 8 TB Flash

RL400: 3xS812LC, 30 cores, 384 GB RAM

RL7000: 3xS822LC, 60 cores, 768 GB RAM, 6TB Flash

RL14000: 3xS822L, 60 cores, 1536 GB RAM, 12TB Flash

31

Customer Example : Redis on Flash

• Genome dataset: 31TBs of raw data

• Optimized data set through encodingand using Redis Hashes

• Resulting data runs high speed analyses with 55GB of RAM and 4.5TB of Flash

• 97% annual savings compared to a pure RAM solution

Redis on RAM Redis on Flash

RAM Size 5TB 0.5TB

Flash size N/A 4.5TB

Serverson AWS :

21x r3.8xlarge on P8:

2x s822 LC

1yr costs $489,333 $15,677

P8 savings 97%

32

Comparison : P8 vs DellSpecs IBM Power8 Dell Power Edge

Model S822 LC Dell PowerEdgeR820

CPU 20 cores, 160 threads(vcores)@2.92 GHz

2x Intel® Xeon® Processor E5-4657L v2; 24 core/48 threads (vcores) @ 2.4Ghz

RAM Size 256GB 256GB

Flash Size 2TB 2TB

Flash IOPS 700k 300k

Network 2x10 Gbps 2x10 Gbps

Price w/o Flash $17,515 $23,033

Price w Flash $23,515 $26,033

• Higher number of virtual

cores on equivalent

boxes

• Higher throughput from

IBM Flash

• 24% lower cost

33

Power 8 Outperforms on RAM

Redis on RAM IBM POWER8 Dell Power Edge

Price w/o Flash $17,515 $23,033

# of shards (dedicated)80 12

$/shard (dedicated)$218.94 $1919.42

# of shards (multi-tenant)320 48

$/shard (multi-tenant)$54.73 $479.85

Max throughput (ops/sec) at sub-msec latency

4,000,000 2,400,000

$/transaction/sec$0.004 $0.01

IBM Throughput % Gain67%

IBM Cost Savings55%

• Higher core count

translates to higher

number of Redis shards,

each capable of handling

very high throughput

• IBM Power 8 delivers

67% higher throughput at

55% lower cost

34

Power 8 Outperforms on Flash

Redis on RAM IBM Power8 Dell Power Edge

Price with Flash $23,515 $26,033

Max throughput (ops/sec) at sub-msec latency

200,000 66,000

$/transaction/sec $0.12 $0.39

IBM Throughput % Gain200%

IBM Cost Savings70%

• More efficient IBM Flash

delivers 200% higher

throughput at 70% lower

cost!

35

Redis Labs Appliances

RL400 – for OLTP use cases 384GB RAM, no Flash, 30 cores

Manufactured by IBM, private-labelled by Avnet

RL7000 – for OLTP/OLAP use cases 768GB RAM, 6TB Flash, 60 cores

RL14000 – for OLAP use cases 1,536GB RAM, 12TB Flash, 60 cores

36

Next Steps

Learn more about Redis for Transactions & Analytics by contacting expert@redislabs.com

To learn more about Redis on Power Systems visit http://ibm.co/2blGNqY or call 1-866-872-3902 (Code: Power)

THANK YOU!