Евгений Курпилянский "Индексирование поверх...

Indexing Cassandra data in SQL-storage

Kurpilyansky Eugene

SKB Kontur

December 9th, 2013

What do we want?

Suppose, we want to store objects of di�erent types in

Cassandra.

Any object has a primary string key.

Cassandra is well-suited for using it as key-value storage.

But we usually want to search among all objects of same type

by some criterion.

Results of searching must be consistent and re�ect current

state of database.

How can we implement storage which satis�es these

requirements?

What do we want?

Cassandra.

by some criterion.

state of database.

requirements?

What do we want?

Cassandra.

by some criterion.

state of database.

requirements?

What do we want?

Cassandra.

by some criterion.

state of database.

requirements?

What do we want?

Cassandra.

by some criterion.

state of database.

requirements?

Using native Cassandra indexes

We can use native Cassandra indexes.

Advantages

There is no need to support additional storage.

Disadvantages

Every custom query may require new CF-structure for

e�ective searching.

SQL-indexes are more e�cient than Cassandra's indexes.

There exist a lot of complex indexes (e.g. full-text search

indexing).

Advantages

Disadvantages

indexing).

Advantages

Disadvantages

indexing).

Advantages

Disadvantages

indexing).

Advantages

Disadvantages

indexing).

Using synchronization with SQL-storage

Main idea

Run IndexService application which is synchronizing data in

SQL-storage with data in Cassandra (constantly,

in background thread).

To perform a search we should make a query to IndexService

which will return the search result after �nishing SQL-storage

synchronization process.

Main idea

Run IndexService application which is synchronizing data in

SQL-storage with data in Cassandra (constantly,

in background thread).

To perform a search we should make a query to IndexService

which will return the search result after �nishing SQL-storage

synchronization process.

Implementation of EventLog

Create event log

One event per one write-request or delete-request.

Event log sorted by time of event.

Create event log

One event per one write-request or delete-request.

Event log sorted by time of event.

Synchronizing SQL-storage with Cassandra

string EventId;

long Timestamp;

string ObjectId;

interface IEventLog

void AddEvent(Event event);

IEnumerable<Event> GetEvents(long fromTicks);

New implementation of IObjectStorage

Before writing or deleting objects call method

IEventLog.AddEvent.

string EventId;

long Timestamp;

string ObjectId;

interface IEventLog

IEventLog.AddEvent.

string EventId;

long Timestamp;

string ObjectId;

interface IEventLog

IEventLog.AddEvent.

EventLog.AddEvent(Event event)

Create column:

ColumnName = event.Timestamp + ':' + event.EventId

ColumnValue = event

EventLog.GetEvents(long fromTicks)

Execute get_slice from exclusive column for one row.

Create column:

ColumnValue = event

Execute get_slice from exclusive column for one row.

We should split all event log into rows using

PartitionInterval to limit size of rows.

PartitionInterval is some constant period of time (e.g.

one hour, or six minutes).

We should split all event log into rows using

PartitionInterval to limit size of rows.

PartitionInterval is some constant period of time (e.g.

one hour, or six minutes).

Create column:

RowKey = event.Timestamp / PartitionInterval.Ticks

ColumnValue = event

Execute get_slice from exclusive column for one or more rows.

Implementation of IndexService

IndexService

It has a local SQL-storage (one storage per one service replica).

There is one SQL-table per one type of object.

There is one speci�c SQL-table for storing times of last

synchronization for each type of object.

There is one background thread per one type of object, which

is reading event log and updating SQL-storage.

For executing incoming SQL-query, we can use data from

SQL-storage and a little range of events.

IndexService

Periodic synchronization action

Set startSynchronizationTime = NowTicks.

Find all events which should be processed.

Process these events: update SQL-storage and keep

unprocessed events (they should be processed on the next

iteration).

Update time of last synchronization to

startSynchronizationTime in SQL-storage.

iteration).

ProcessEvents(Event[] events)

This function actualizes values of related objects in SQL-storage.

Remember, that we update object after creating an event.

So, we can not process some of events at the moment, because

correspoding object isn't updated yet.

ProcessEvents(Event[] events)

This function actualizes values of related objects in SQL-storage.

Remember, that we update object after creating an event.

So, we can not process some of events at the moment, because

correspoding object isn't updated yet.

Event[] ProcessEvents(Event[] events)

This function actualizes values of related objects in SQL-storage

and returns events, which have not been processed.

How will this function be implemented?

For every event we should analyze corresponding objects from both

Cassandra and SQL-storage.

Example 1

event = {Timestamp: 2008}

cassObj = {Timestamp: 2008, School: 'USU'}

sqlObj = {Timestamp: 2005, School: 'AESÑ USU'}

What should we do?

Example 1

Write cassObj in SQL-storage and mark event as processed.

Example 1

Example 2

What should we do?

Example 1

Example 2

Timestamp of event is greater than timestamp of cassObj.

Probably, it needs to wait for updating of object.

Example 1

Example 2

Timestamp of event is greater than timestamp of cassObj.

Probably, it needs to wait for updating of object.

Write cassObj in SQL-storage and mark event as unprocessed.

Example 3

cassObj is missing

sqlObj is missing

What should we do?

Example 3

cassObj is missing

sqlObj is missing

Probably, that event corresponds to the creation of object.

Example 3

cassObj is missing

sqlObj is missing

Mark event as unprocessed.

Example 3

cassObj is missing

sqlObj is missing

Example 4

cassObj is missing

sqlObj = {Timestamp: 2012, School: 'UFU'}

What should we do?

Example 3

cassObj is missing

sqlObj is missing

Example 4

cassObj is missing

Two cases are possible:

1 That event corresponds to the deletion of object.

2 That event corresponds to the creation of object. sqlObj is

not missing, because there were two operationsin a row: delete

and create.

Example 4

cassObj is missing

Two cases are possible:

1 That event corresponds to the deletion of object.

2 That event corresponds to the creation of object. sqlObj is

not missing, because there were two operationsin a row: delete

and create.

Delete sqlObj from SQL-storage and mark event as unprocessed.

Read objects, which occured in these events, from Cassandra and

SQL-storage (some of them can be missing).

For each (event, cassObj, sqlObj) do

If cassObj is not missing

Save cassObj in SQL-storageIf event.Timestamp <= cassObj.Timestamp

then mark event as processed;

else mark event as unprocessed.

else (i.e. cassObj is missing)

Delete sqlObj from SQL-storage if it's not missing.

Return events which has been marked as unprocessed.

iteration).

What events should we use as arguments in ProcessEvents

function?

Of course, all unprocessed events from previous iteration.

Also all new events, i.e. IEventLog.GetEvents(fromTicks).

What is fromTicks?

fromTicks = lastSynchronizationTime?

No. Unfortunately, any operation with Cassandra can be

executed for a long time.

This time is limited by

writeTimeout = attemptsCount · connectionTimeout.

We should make undertow back, otherwise we can lose some

events.

fromTicks = lastSynchronizationTime - writeTimeout

function?

What is fromTicks?

events.

function?

What is fromTicks?

events.

function?

What is fromTicks?

events.

function?

What is fromTicks?

events.

function?

What is fromTicks?

events.

function?

What is fromTicks?

events.

Executing search request

Advantages.

Scalability.

Availability.

Fault tolerance.

Sharding.

Advantages.

Scalability.

Availability.

Fault tolerance.

Sharding.

Questions

Thank you for your attention. Any questions?

Евгений Курпилянский "Индексирование поверх...

Technology

additional storage

native cassandra indexeswe

keyvalue storage

sqlstorage main ideamain

cassandras indexes

text search indexing

lot of complex indexes

search result