kdb database (epam tech talks, sofia, april, 2015)
TRANSCRIPT
1
KDB Database
Martin Toshev 25.04.2015
2
Agenda
KDB Database Overview
Q Lang Basics
3
Agenda
KDB vs classical RDBMS
KDB Java API
KDB Use Cases
4
KDB Database Overview
5
KDB database overview
KDB:
• an in-memory column-based database
• can be used to handle large volumes of real-time or historical data
• provides OLTP and OLAP capabilities
6
KDB database overview
KDB:
• provides much of the same functionality of an RDBMS
• supports SQL and KSql (for column-based queries)
7
KDB database overview
Used by almost any large financial institution …
• Goldman Sachs
• Morgan Stanley
• J.P. Morgan
• Deutsche Bank
• Commerzbank AG
8
KDB database overview
KDB+:
• 64bit release of KDB
• OLTP: 100 000 – 1 000 000 records per second per CPU
• OLAP: 1 000 000 – 10 000 000 records per second per CPU
• Backed by the Q general purpose programming language
9
KDB database overview
KDB+:
• multi-threaded and multi-process (scales vertically)
• can be deployed in a distributed environment (scales horizontally)
• provides multiple APIs for C++, Java, Perl, Python …
• and provides support for JDBC …
10
Q Lang Basics
11
Q Lang Basics
Q language:
• vector processing language
• allows for complex calculations across a large volume of data
• merges the K and ksql languages
12
Q Lang Basics
Q language:
• Interpreted (a mixture of APL and Scheme)
• simple rules of precedence (right -> left evaluation of all expressions)
• functions are first-class citizens (can be assigned to variables)
13
Q Lang Basics
Q language data types:
• Integer
• floating point
• char
• symbol
• lists
• dictionaries
• time • datetime • date
• others …
14
Q Lang
Demo
15
KDB vs classical RDBMS
16
KDB vs classical RDBMS
KDB is column-oriented while RDBMS are row-oriented
Both support SQL for querying the database
17
KDB vs classical RDBMS
Classical RDBMS are in control of the persistence
In KDB+ the user is in control of the persistence
18
KDB vs classical RDBMS
No concept of transactions in KDB+
Queries in KDB+ are always SERIALIZABLE in terms of
isolation levels
19
KDB vs classical RDBMS
KDB provides utilities for dealing with time-series data:
• time joins (af – asof join, wj – window join)
20
KDB Java API
21
KDB Java API
Extremely simple …
A single class c that represents a KDB+ connection
22
KDB Java API
c.ks() – for executing an operation without waiting for a
response
c.k() – for execution an operation and waiting for a response
c.k() – for waiting for a message from the KDB+ server
(naming is somewhat dummy … )
23
KDB Java API
// getting the 'tbl' table
c connection = new c("localhost", 5000);
Flip tbl = (Flip) connection.k("tbl");
for(String key : tbl.x) {
System.out.println(key + " " +
tbl.at(key).getClass().getSimpleName());
}
24
KDB Java API
// inserting a record in the 'tbl‘ table
connection.ks("`tbl insert (`x;10)");
25
KDB Java API
Demo
26
KDB Use Cases
27
KDB Use Cases
Capturing, storing and analyzing massive volumes of data … (kdb+tick)
Kdb+tick tickerplant
Market data feeds
Other feeds
KDB+ log files
Kdb+tick rdb
Java/C/ Python
Kdb+tick hdb
28
KDB Use Cases
Analyzing FIX messages for post trade analytics and compliance (kdb+tick)
Kdb+tick tickerplant
Market data feeds
Other feeds
KDB+ log files
Kdb+tick rdb
Java/C/ Python
Kdb+tick hdb
OMS (Order Management
System)
29
KDB Use Cases
Importing and analyzing massive volumes of data from an exchange … (kdb+taq)
Kdb+tick
taq
NYSE
30
Thank you
31
Resources
Wikipedia’s entry on KDB
http://en.wikipedia.org/wiki/KDB
KDB wiki
http://code.kx.com/wiki/Main_Page
KDB-tick overview
http://kx.com/papers/Kdb+tickDiagram.pdf
KDB guides
http://www.timestored.com/kdb-guides/
32
Resources
K language
http://www.math.bas.bg/bantchev/place/k.html
KDB for Gods
http://www.firstderivatives.com/lecture_series.asp
KDB+ function reference
http://code.kx.com/wiki/Reference
33
Resources
Kdb+ Database and Language Primer
http://kx.com/q/d/primer.htm
KDB+ Database Reference Manual
http://kx.com/q/d/kdb+1.htm
Q language reference manual
http://kx.com/q/d/q1.htm
Abridged KDB+ Database Manual
http://kx.com/q/d/kdb+.htm
Abridged Q Language Manual
http://www.kx.com/q/d/q.htm
34
Resources
Kdb database tutorial
http://www.thalesians.com/finance/index.php/Knowledge_Base/Databases/Kdb
A shallow introduction to the K language
http://www.kuro5hin.org/?op=displaystory;sid=2002/11/14/22741/791
K as a prototyping language
http://www.cs.nyu.edu/courses/fall02/G22.3033-007/kintro.html