Kx Technology – a Big Data SolutionKx Community Zurich Meetup

Kyra CoyneNovember 2016

2

• Global company, division of First Derivatives plc (listed on LSE)• Large user community• Widely adopted in financial services over two decades• Software & industry solutions, consulting and implementation services

Known for:• Processing and analysis of large volumes of real-time and historical time series data• Extreme performance (low latency)• Integrates with and co-exists with other technologies• Ability to scale without requiring significant infrastructure

About Kx

3

About the Technology• Integrated in-memory,

columnar database & programming system

• Streaming, real-time and historical data

• Map-Reduce built-in

• Native time-seriesfunctions

• Light-weight (~500kb)

• Standard OS & hardware

• Extreme Performance

4

Kx Technology

• Integrated columnar database & programming system

• Streaming, real time and historical data

• Built for massive data volumes

• In-database analytics

• Parallelism

• Compression

WHAT IS Kx TECHNOLOGY

KEY FEATURES

• Interpreted

• Event-driven

• Functional

• Array / Vector

• Query

• Time-series

q PROGRAMMING LANGUAGE

WHY Kx TECHNOLOGY? HIGH PERFORMANCE, LOW LATENCY • We are fast not only due to data architecture, our

native programming language, q, runs inside the database not in separate processes with costly data passing.

POWER• We are one of the few fully 64 bit databases

and unique in having time as a native type, with nanosecond resolution and a full set of operations over time.

QUICK TO DELIVER • Unlike many compile-link and run approaches q is

dynamic allowing much shorter development and deployment cycles

5

The Big Data Landscape

4

• 20 year track record; mission critical systems

• Streaming, real-time and historical data

• Processing and analyzing data in microseconds

• Hundreds of millions of transactions per second

• Terabytes to petabytes

• Trusted globally by largest institutions

6

COST• Fully transparent costing models • Reduction in personnel, training, hardware and facilities costs

for clients

SECURITY• Robust, high performance infrastructure• Highly Secure• Comprehensive disaster recovery and business continuity planning

SERVICE • Best Practice and processes• Large pool of highly skilled engineers

RISK• Scalable model to respond to changes in demand

7

• Large Canadian Utility (IESO)•Meter Data Management System• Processes 4.7 million meters• 120 million meter readings per day• Oracle RDBMS has 300+ billion records• Could not accommodate demand for

analytics

Downstream Case Study

8

Downstream Pattern Case Study (before)

Meter Data Management

Distributors

CIS / AMISystem

Consumers

Billing Statement

Web Presentment

Meter Data

MasterData

Meter Data

Meter Data

Billing Request

Billing Response

Web Service Request

Web Service Response

Reports

Master Data

Before

9

Downstream Pattern Case Study (after)

Meter Data Management

Distributors

CIS / AMISystem

Consumers

Billing Statement

Web Presentment

Meter Data

Meter Reads Retrieval

Web Services

MasterData

Meter Data

Meter Data

Billing Request

Billing Response

Web Service Request

Web Service Response

Reports

Master Data

Transform & Load

Bulk Data Extract

Change Data

Capture(Real Time)

Initial Extract

Intra-Day

Extract

Queries &Visualisation

Kdb+Database

Kx TechnologiesAfter

10

Downstream Pattern Results

Service PreviousSolution

KxSolution

Web-services response time 30 sec 30 ms

Number of requests 30/sec 20,000/sec

Ad-hoc queries, aggregations, bulk extracts Hours-Days Second-Minutes

Service Availability 20x7, 6 days/week 24x7, 7 days/week

11

Downstream Pattern

Pros• Maintains investment in

existing system• Rapid implementation of Kx

technology• Low risk or impact on existing

system• Functionality and availability

improvedCons• Added storage and possibly

licensing costs• Updates must flow through

existing system• Doesn’t address streaming

Data Feed

Existing Systemof

Record

Time series & Master

Data in kdb+

Kx Technologies

Ad hocQueries

Analysis

Dashboards

Time series data

Real-time or scheduled replication

Query results

12

From Technology to Solutions

Solution

Kx Tools

Kx Platform

13

Kx Use CasesBUSINESS USE CASES APPLICATION

Real-time Analytics Tick-capture and streaming data is analysed and enriched in real-time to produce live indicators of current conditions for further action.

Quantitative Research Run trade, quote and fundamental analysis on large datasets and produce trading indicators faster.

Risk Management Intra-day, pricing, credit, exposure and P&L alerts with visual tools, including heat maps and OLAP drill downs to monitor activity.

Market Surveillance Implement trading control alerts related to in-house and regulatory requirements, as well as the generation of planning reports.

Depth-of-Book Analysis Create real-time depth-of-book views for any instrument across thousands of symbols. Build order books from disk in sub second.

Network and Hardware Management

Manage multi-server distributed environments from a single dashboard. Monitor the health of thousands of processes and servers across plants spread throughout multiple regions.

Internet of Things (IoT) Real-time capture and processing of data generated by sensors in machines, homes, cars, smart meters, mobile phones and other devices.

14

Kx Performance

15

Kx Performance

16

Kx Performance

Legend

• DNF = Did Not Finish

• RAM is memory used for queries

• Query times are in milliseconds

kdb+ is 10 to 100 times faster than other colstore (vertical, big3accel, hadoop / impala / parquet, ..)

kdb+ is 100 to 1000 times faster than the rowstore (postgres, big3rdbms, mongodb, spark, ..)

17

Reference Architecture

| Kx Systems

Server 12

n

RDB 1 RDB n RTE

GW

Server 12

n

FH 1 FH 2 FH n

TP

Real-time Data

Server 12

n

PDB

HDB 1 HDB n

ClientLegend

Data Flow

Queries & Results

Data Persistence

Acronyms

FH = Feed HandlerTP = TickerplantRDB = Realtime DBRTE = CEP EngineGW = GatewayHDB = Historic DBPDB = Persisting DB

18

Kx Stream

19

Engineering for Performance

The worlds leading time series database, specifically designed for handling massive data volumes and real-time streaming analytics.

• To improve search performance and data consumption we apply sharding, where data is split between multiple servers.

• Stream for Kx offers inbuilt horizontal scaling across all data micro services.

• Horizontal scalability is applied at the point of data capture and data querying.

• In-built Map Reduce means that results are virtually instantaneous without the additional overhead of defining unique aggregation logic.

• For increased capacity, we use replication. This is the process of mirroring our data-set.

20

How do we use our technology?

• Simplify both real-time & historical data using one powerful enterprise platform.

TOOLS for Kx - HARNESS THE POWER OF DATA

Kx - CORE TECHNOLOGY• Streaming analytics, in-memory compute and

database technology, providing a full application server with a powerful functional scripting language.  

Kx SOLUTIONS - BUILD POWERFUL BUSINESS INSIGHTS• Our tools are used to accelerate implementation of proven solutions for complex problems.

OUR PEOPLE - CHALLENGE US WITH YOUR UNIQUE PROBLEMS• Our engineers develop, deploy and support solutions for virtually any problem involving massive amounts of data

21

Our Industry Solutions

Stream for Kx Tools to rapidly develop and deploy streaming real-time and historical analytics

Kx Technology

Kx for Algo

Build, Test, and deploy

Algorithmic Trading Strategies

Kx for Pharma

Kx for Analytics

Real Time and Historical Market

Analytics

Kx for Sensors

Dashboards for Kx

Kx for Surveillance

Real time market monitoring Surveillance

Workflow

IMS patient record analytics,

manufacturing, clinical research

Smart meters, utilities,

geolocation, customer analytics

22

Kx® and kdb+ are registered trademarks of Kx Systems, Inc., a subsidiary of First Derivatives plc

Resources• Free 32bit download version: http://kx.com/software-download.php

• Kx Wiki: http://code.kx.com/wiki/Main_Page

• Google Group: https://groups.google.com/forum/#!forum/personal-kdbplus

• Kx Github: http://kxsystems.github.io/

• STAC benchmarks: https://stacresearch.com/kx

• Kx Meetups: http://kx.meetup.com/

Kyra Coyne

kcoyne@firstderivatives.com

+4917659883653