introduction to chronicle (low latency persistence)

(c) Higher Frequency Trading

Who am I? Australian living in UK. Three kids 5, 8 and 15 Five years designing, developing and supporting HFT

systems in Java My blog, “Vanilla Java” gets 120K page views per month. 3rd for Java on StackOverflow Lead developer for OpenHFT which includes Chronicle

and Thread Affinity.


Introduction to Chronicle

Micro second latency persistence

Higher Frequency Trading(c) Peter Lawrey


What is Chronicle?

Very fast embedded persistence for Java.

Functionality is simple and low level by design


Where does Chronicle come from

Low latency, high frequency trading

– Applications which are sub 100 micro-second external to the system.



High throughput trading systems

– Hundreds of thousand of events per second



Modes of use

– GC free

– Lock-less

– Shared memory

– Text or binary

– Replicated over TCP

– Supports thread affinity


Is there a free version?

It is open source and free with an Apache 2.0 license.

You can pay for training and consulting


Use for Chronicle

Synchronous text logging Synchronous binary data logging


Use for Chronicle

Messaging between processesvia shared memory

Messaging across systems


Use for Chronicle

Supports recording micro-second timestamps across the systems

Replay for production data in test


Writing to Chronicle

IndexedChronicle ic = new IndexedChronicle(basePath);Excerpt excerpt = ic.createExcerpt();

for (int i = 1; i <= runs; i++) { excerpt.startExcerpt(17); excerpt.writeUnsignedByte('M'); // message type excerpt.writeLong(i); // e.g. time stamp excerpt.writeDouble(i); excerpt.finish();}ic.close();


Reading from ChronicleIndexedChronicle ic = new IndexedChronicle(basePath);ic.useUnsafe(true); // for benchmarksExcerpt excerpt = ic.createExcerpt();for (int i = 1; i <= runs; i++) { while (!excerpt.nextIndex()) { // busy wait } char ch = (char) excerpt.readUnsignedByte(); long l = excerpt.readLong(); double d = excerpt.readDouble(); assert ch == 'M'; assert l == i; assert d == i; excerpt.finish();}ic.close();


How does it perform

With one thread writing and another reading

* Chronicle 2.0 -XX:+AggressiveOpts -Xmx32m

Tiny4 B

Small16 B

Medium64 B

Large256 B

tmpfs 77 M/s 57 M/s 23 M/s 6.6 M/s

ext4 65 M/s 35 M/s 12 M/s 3.2 M/s


How does it recover?

Once finish() returns, the OS will do the rest.

If an excerpt is incomplete, it will be pruned.


Cache friendly

Data is laid out continuously, naturally packed. You can compress some types. One entry starts in the next byte to the previous one.


Consumer insensitive

No matter how slow the consumer is, the producer never has to wait. It never needs to clean messages before publishing (as a ring buffer does)

You can start a consumer at the end of the day e.g. for reporting. The consumer can be more than the main memory size behind the producer as a Chronicle is not limited by main memory.


How does it collect garbage?

There is an assumption that your application has a daily or weekly maintenance cycle.

This is implemented by closing the files and creating new ones. i.e. the whole lot is moved, compressed or deleted.

Anything which must be retained can be copied to the new Chronicle


Is there a lower level API?

Chronicle 2.0 is based on OpenHFT Java Lang library which supports access to 64-bit native memory.● Has long size and offsets.● Support serialization and deserialization● Thread safe access including locking


Is there a higher level API?

You can hide the low level details with an interface.



There is a demo program with a simple interface.

This models a “hub” process which take in events, processes them and publishes results.



The interfaces look like this

public interface Gw2PeEvents {

public void small(MetaData metaData, SmallCommand command);

}

public interface Pe2GwEvents {

public void report(MetaData metaData, SmallReport smallReport);

}

On this laptop, it performs around 400K msg/sin and out, persisted.


What does Chronicle need

More documentation

More tutorials

More use case examples


What is planned

Lower overhead per message 80 ns → 15 ns.

Low latency XML and FIX parser and writer

Central Order Book engine.


Q & A

introduction to chronicle (low latency persistence)

Technology

c higher frequency trading

chronicle messaging

chronicle indexedchronicle

vanilla java

hft systems

shared memory text

shared memory messaging

modes of use gc free