real time big data in the financial space

Real-time Big Data in Financial Services

Perry Krug – Couchbase Principal Solutions ArchitectBrian Hopkins – Forrester Research Analyst

1

Source: “Connected Cows?” Joseph Sirosh, Microsoft at Strata San Jose 2015

Image source: Siloam Springs Veterinary Clinic (http://siloamspringsvet.com/)

The tale of the connected cow

It is the insight that matters.

Image source: Tsevo (http://www.tsevo.com/)

© 2015 Forrester Research, Inc. Reproduction Prohibited 4

Big data promises insights discovery

65% of firms will be

using big data

analytics to optimize

digital experiences.

56% of firms will

have implemented

Hadoop.

38% of firms will

have spent more

than $10 million on

data and analytics.

Base: 3,005 business and technology decision-makers Source: Forrester’s Global Business Technographics® Data And Analytics Survey, 2015


The age of the customer is the driving force

More than 50% of

technology project

spending will focus on

age of the customer

objectives in 2015.


The data hub circa 2012

› Source: June 12, 2013, “Deliver On Big Data Potential With A Hub-And-Spoke Architecture” Forrester report


The data hub circa 2015

› Source: March 25, 2015, “Boost Your Business Insights By Converging Big Data And BI” Forrester report


Data is a means; effective action is the end

Right Data

Effective Actions

Digital Insights

X Volume and variety are bottlenecks

X Testing and finding the right

ones is time consuming

X Deploying into software is painful


You will need a digital insights architecture

Insights-driven

applications

• Sales and marketing

• Operations

• Product design

• . . . lots more

Insight

Fabric

• Services framework

• Federation and virtualization

• Big data management

• Cloud services

Data feed

management

Execution

engines

Insight

discovery and

implementation

tools

Collaboration

and governance

tools

Source: April 27, 2015, “Digital Insight Is The New Currency Of Business” Forrester report


Financial services insight trends and challenges

Trends

› Age of the customer and

regulations forcing change

› Engagement data piles onto

tons of SoR data

› From EDW to discovery and

text analytics

› Re-platforming for real time

Challenges

› Real time customer

engagement

› Real time fraud loss prevention

› Improving the cost for legacy

systems performance

What is Couchbase

©2014 Couchbase Inc.

24x365

What makes Couchbase unique?

12

Performance & scalability leader

Sub millisecond latency with high throughput; memory-centric architecture

Multi-purpose

Simplified administration

Easy to deploy & manage; integrated Admin Console, single-click cluster expansion & rebalance

Cache, key value store, document database, and local/mobile database in single platform

Always-on availability

Data replication across nodes, clusters, and data centers

Enterprises choose Couchbase for several key advantages


Couchbase provides a complete Data Management solution

13

High availability cache

Key-value store

Document database

Embedded database

Sync management

Multi-purpose capabilities support a broad range of apps and use cases


Security with Couchbase

Administrative Security: Admin user (full access) vs. Read-only user (monitoring, developer access) SSL encryption to REST API and Web UI HTTP access log

Data Security: Applications connect via SASL with single user/pass Data-at-Rest encryption via partnership with Vormetric SSL encryption for over-the-wire

Coming in 4.0: LDAP/Kerberos integration Extensive administrative action auditing


Big Data = Operational + Analytic (NoSQL + Hadoop)

15

Online

Web/Mobile/IoT apps

Millions of customers/consumers

Offline

Analytics apps

Hundreds of business analysts


Couchbase + Big Data

New Data Stream

Merged View

All DataPrecompute

Views (Map Reduce)

Process Stream

Incremental Views

Partial Aggregate

Partial Aggregate

Partial Aggregate

Real-Time Data

BatchRecompute

Batch Views

Real-Time Views

Real-TimeIncrement

Merge

Batch Layer

Serving Layer

Speed Layer

Couchbase HadoopConnector

Couchbase and Financial Services


Real Time Big Data

18

ObjectiveDrive revenue, customer satisfaction, and operational efficiency by leveraging insights from big data analytics in real time

Business requirements

Manage massive data volumes at high speed

Store and manage numerous and changing data types

Export/import data to/from analytics platformsThe Couchbase Solution

Push-button scalability – fast, easy and inexpensive to scale to any size

Integrated cache – enables fast performance and high throughput

Flexible JSON data model – easily adapts new data types and attributes on the fly

Real time Hadoop integration via in-memory streaming – easily export data and import analytics results

Technical requirements

Scalability and throughput

Data model flexibility

Integrate with Hadoop


User Activity Tracking and real-time analyticsObjectives & Challenges

Provide business users with real time reports and visualizations of user interaction data

Collect web and mobile clickstream in real time

Integrate with other big data technologies (Hadoop and Storm)

Provide views of data across multiple dimensions (e.g., time, location, browser and device types)

19

130m+ active accounts, in 190+ countries, 25 currencies

10TB data

1B documents

Solution

Deploy Couchbase Server to capture, store, and process real time web data

Ingests data (via Storm) from multiple inputs, including mobile, web, and other services, storing data as JSON documents

Integrates with Hadoop to pass data for additional offline analytics

Generates views for business users in under 1 minute, based on 10-minute data collection intervals

The Couchbase AdvantageReal time performance, easy integration with Storm and Hadoop


User Activity Tracking and real-time analytics

20

Couchbase Solution Couchbase Server deployed to capture, store, and process real time

web data Ingests data (via Storm) from multiple inputs, including mobile, web,

and other services, storing data as JSON documents Integrates with Hadoop to pass data for additional offline analytics

Results Consistent low latency (sub 10-msec response) High availability enabled by distributed caching and XDCR Views for business users are generated in under 1 minute, based on

10-minute data collection intervals


User Activity Tracking and real-time analytics

21

Real-Time Big Data


Fraud Detection

24

ObjectiveIncrease profitability, reduce risk, and comply with regulations in real-time by analyzing user, transaction, and contextual data

Business requirements

Update frequently changing data and data types -customer data ,account data, detection rules

Provide real time responsiveness

Process very high volume of interactions

The Couchbase Solution

Flexible JSON data model – easily adapts new data types and attributes on the fly

Integrated cache – provides real time responsiveness and high throughput processing

Push-button scalability – fast and easy to meet growth requirements

Technical requirements

Data model flexibility

Low latency

High throughput


Fraud Detection with FICOObjective & Challenges

Capture transactions, store card / account profiles, customer profiles & user defined variables with sub-msec latency and high throughput

Growing number of accounts, cards and customers means more data needs to be tracked and faster latencies are required

Relational systems unable to scale to the required throughput

HA / DR solutions not streamlined – need custom development

25

Falcon

#1 Fraud Detection platform in the world

65% of worlds credit / debit cards scored by Falcon

Solution

Use Couchbase as the “profiling store” and replace relational database

Each Falcon customer has 100’s of millions of card and / or account profiles that can easily be stored and updated based on consumer’s real time activity

Neural networking algorithms run on Couchbase and access data as key-value pairs. Memory-first architecture allows <1ms responses.

Complete HA / DR solution delivers 24x365 application uptime

The Couchbase AdvantageMemory-first architecture means high throughput, all with click-button scalability


Fraud Detection with Couchbase at Wells Fargo

26

Couchbase Solution Use Couchbase as the “profiling store” and replace relational database Each Falcon customer has 100’s of millions of card and / or account

profiles that can easily be stored and updated based on consumer’s real time activity

Results Complete HA / DR solution delivers 24x365 application uptime Memory-first architecture allows <1ms responses. Neural networking algorithms run on Couchbase and access data as

key-value pairs


Fraud Detection with Couchbase at Wells Fargo

27

Real-Time Fraud Detection


Improving Legacy Systems through caching

30

The problemProvide low latency and high throughput access to a variety of data types. Alleviate load on backend systems.

Application Requirements

Lots of users accessing different datasets

Data in varying formats: HTML, JSON, protobuf

High read performance

Uptime critical

Legacy systems are hard and expensive to scale

The Couchbase Solution Based on memcached = fast!

Replicated and persistent with auto-failover

Fully distributed and clustered with “push button” scaling: easy, inexpensive

Support for binary and JSON data types

Challenges with other caching technologies

Complicated to setup and monitor

Not persistent

Restriction of supported data types

Not truly distributed or clustered (i.e. ehcache)

Poor performance under load


Caching @ ExperianThe problem

Massive and spikey user traffic to small bits of data supporting web experience

Mainframes are expensive to scale

New applications constantly coming online

Need to handle spikes in traffic

31

Experian.com

Provide fast access to credit information

The solution

Deploy Couchbase Server as standardized distributed caching layer

Compatible with memcached, highly optimized for latency and throughput

Shared nothing, replicated and persistent for reliability

Support for JSON as well as any binary data type

Shared-nothing, replicated and persistent architecture

The Couchbase AdvantageMassive speed and scale that’s easy to manage


Caching @ Experian

32

User Requests

Cache Misses and Write Requests

Mainframe

ApplicationLayer

CouchbaseDistributed Cache

Read-Write Requests

Improving Legacy Systems

Thank you

Perry Krug – [email protected] Hopkins - [email protected]

real time big data in the financial space

Software

data hub

data centersenterprises

big data analytics

big data potential

changeengagement data

bi forrester report

architecture forrester

publicscheduled image