mariadb pres at lemug

© MariaDB Corporation Ab

MariaDBStorage Engines

Serge Frezefond, Cloud Solution Architect

[email protected]

@sfrezefond

http://serge.frezefond.com

* *23/02/2015 1

mailto:[email protected]


Agenda

• TokuDB

• Spider

• CONNECT

• Conclusion

23/02/2015 2


TokuDB - Better Performance

• TokuDB® uses Fractal Tree® technology:– Internal nodes are similar to B-trees, but they also have

message buffers

– Large block size (4M) enables better compression performance and range queries.

– Basement nodes support point queries

o Default size 128K

– Optimal I/O utilization:

o Reads: highly compressed data

o Writes: aggregation of multiple operations + high compression.


TokuDB - Fractal Tree Indexes

Each node has

pivots & Buffers

Buffers fill

as updates arrive


Fractal Tree Indexes

Each node

has pivots & BuffersBuffers fill

as updates arrive

Flush a buffer

when it fills

A flush might take an I/O,

but it does lots of useful work

More changes per write ➔

fewer changes for same write load ➔

less SSD wear


Better Indexing Improves Performance

•

• High performance during INSERT/UPDATE/DELETE operations


Typical Strategies

• Partitioning

– Adds complexity

– Table scans, when needed, are slower

• Covering indexes

– Secondary keys require additional PK lookup

– Slows down write operations

• InnoDB compression

– Affects performance

• Read-only slaves

– Adds complexity

– Slave delays

• SSD


Using TokuDB

• Partitioning

– Better performance w/ bigger partitions

• CLUSTERING KEY

– Better secondary key insertion

– Less indexes overall

• TokuDB compression

– Always on

– Compression alternatives: quicklz, zlib, lzma

• Read-only slaves

– Upsert / Insert optimization

• SSD

– Compression == better utilization


Spider Storage Engine

• Developed by Kentoku Shiba

• Storage Engine „partitions“ tablesacross multiple Database Server instances

• Based on Partitions with integrated Sharding

• Virtual view on tables distributed across Instances

• Supports XA transactions

• Transactional Storage Engine

• Provides Scale-Outin Combination with HA

9

Application

Customer

A-H

I-P

Q-Z

Spider

Customet

A-H

Shard 1

Customer

I-P

Shard 2

Customer

Q-Z

Shard 3


Sharding using Spider

10

Backend 2

Application

Spider

Backend 3

T1

P1

P2

P3

T1 P2 T1

P3

Backend 1

T1

P1

XA 2 PC


Sharding with Spider and HA

11

Backend 2

Application

Spider

Backend 3

T1

P1

P2

P3

T1 P2 T1

P3

Backend 1

T1

P1

XA 2 PC

P2

P3

P1


Spider Storage EngineInstallation

• Installation

• Spider will be shown as active Storage Engine

12

mysql -uroot -p < /usr/share/mysql/install_spider.sql

SELECT engine, support, transactions, xa FROM

information_schema.engines;

+--------------------+---------+--------------+------+

| engine | support | transactions | xa |

+--------------------+---------+--------------+------+

| SPIDER | YES | YES | YES |

| CSV | YES | NO | NO |



• 93 Spider System-Variables will be added

• 4 Spider Status-Values will be added

• More Spider Variables related to tables using CREATE TABLE

• In MariaDB use COMMENT

• In MySQL use CONNECTION14

MariaDB [mysql]> show global variables like 'spider%’;

MariaDB [mysql]> show global status like 'spider%’;


Spider Storage EngineExample without Sharding

• Table definition on Spider Proxy-Node

• Table definition on Backend-Nodes

15

CREATE TABLE spiderfederation(id INT NOT NULL, code

VARCHAR(10), PRIMARY KEY(id))

ENGINE=SPIDER

COMMENT 'host "192.168.56.21", user "backend", password

"backend", port "3306"';

CREATE TABLE spiderfederation(id INT NOT NULL, code

VARCHAR(10), PRIMARY KEY(id))

ENGINE=INNODB;


Spider Storage EngineExample using Sharding

• Table definition on Spider Proxy-Node

16

CREATE TABLE sharding(id INT NOT NULL, code VARCHAR(10),

PRIMARY KEY(id))

ENGINE=SPIDER COMMENT='user "backend", password

"backend", port "3306", table "sharding"'

PARTITION BY RANGE(id)

(

PARTITION p1 VALUES LESS THAN (100000)

COMMENT 'host "192.168.56.21"',

PARTITION p2 VALUES LESS THAN (200000)

COMMENT 'host "192.168.56.22"',

PARTITION p3 VALUES LESS THAN MAXVALUE

COMMENT 'host "192.168.56.23"'

);



• Spider UDFs will be added

• SPIDER_DIRECT_SQL

• Execute SQL on Backend-Server

• SPIDER_BG_DIRECT_SQL

• Execute SQL on Backend-Server

• SPIDER_COPY_TABLES

• SPIDER_FLUSH_TABLE_MON_CACHE• Reset Spider Monitoring Information

17


CONNECT Storage Engine

23/02/2015 18


Goal of the CONNECT Storage Engine :BI on various targets

Most of the data in companies is in variousexternal datasources (many in non relationaldatabase format) :– relational databases: Oracle, SQL Server…

– Dbase, Firebird, SQlite

– Microsoft Access & Excel

– Distributed mysql servers

– DOS,FIX,BIN,CSV, XML

– stored per column...

Not targeted for OLTP2014/02/02 19


The CONNECT Storage Engine

2014/02/02 20

MySQL Server / MariaDB

MyISA

MInnoDB Memory Connect

Federate

dMerge CSV ...

ODBC MySQL XML CSV DIR TBL JSON ...

XML CSV ODBCMySQ

LDIR ...


The CONNECT Storage Engineimplements advanced features

Condition Push down

– Used with ODBC and MySQL to push condition to the target database. Big perf gain

set optimizer_switch='engine_condition_pushdown=on‘

• Support MariaDB virtuals columns

• Support of special columns :

– Rowid, fileid, tabid, servid

• Extensible with the OEM file type

• Catalog table for table metadata(ODBC …)


CONECTFile table type

• DOS,FIX,BIN,FMT,CSV,INI,XML, JSON

• Support virtual tables (DIR)

• Large tables support (>2GB)

• Compression - gzlib format

• Memory file maping

• Add read optimized indexing to files

• Multiple CONNECT tables can be created on the same underlying file

• Indexes can be shared between tables


CONNECT Storage EngineODBC table type

Allow to access to any ODBC datasource.– Excel, Access, Firebird, SQLite– SQL Server, Oracle, DB2

• Supports insert, update, delete and any othercommands

• Multi files ODBC: consolidated monthly exceldatasheet

• Access to ODBC and UnixODBC data sources• WHERE conditions are push to the ODBC source


XML Table Type

<?xml version="1.0" encoding="ISO-8859-1"?>

<BIBLIO SUBJECT="XML">

<BOOK ISBN="9782212090819" LANG="fr" SUBJECT="applications">

<AUTHOR>

<FIRSTNAME>Jean-Christophe</FIRSTNAME>

<LASTNAME>Bernadac</LASTNAME>

</AUTHOR>

<TITLE>Construire une application XML</TITLE>

<PUBLISHER>

<NAME>Eyrolles</NAME> <PLACE>Paris</PLACE>

</PUBLISHER>

<DATEPUB>1999</DATEPUB>

</BOOK>

</BIBLIO>


XML Table Type

create table xsampall (

isbn char(15) field_format='@ISBN',

authorln char(20) field_format='AUTHOR/LASTNAME',

title char(32) field_format='TITLE',

translated char(32) field_format='TRANSLATOR/@PREFIX',

year int(4) field_format='DATEPUB')

engine=CONNECT table_type=XML file_name='Xsample.xml'

tabname='BIBLIO' option_list='rownode=BOOK,skipnull=1';


XMLTable TypeQuery Result

select isbn, subject, title, publisher fromxsamp2;

ISBN SUBJEC TTITLE PUBLISHER

9782212090819 applications Construire une application XML Eyrolles Paris

9782840825685 applications XML en Action Microsoft Press

Can also generate HTML


JSON Table Type

[

{

"ISBN": "9782212090819",

"LANG": "fr",

"SUBJECT": "applications",

"AUTHOR": [

{

"FIRSTNAME": "Jean-Christophe",

"LASTNAME": "Bernadac"

},

… ],

"TITLE": "Construire une application XML",

"PUBLISHER": {

…


JSON Table Type

create table jsampall (

ISBN char(15),

Language char(2) field_format='LANG',

Subject char(32) field_format='SUBJECT',

Author char(128) field_format='AUTHOR:[" and "]',

Title char(32) field_format='TITLE',

Publisher char(20) field_format='PUBLISHER:NAME',

Location char(16) field_format='PUBLISHER:PLACE',

Year int(4) field_format='DATEPUB')

engine=CONNECT table_type=JSON File_name='biblio3.jsn';


JSON Table TypeQuery Result

select title, author, publisher, location from jsampall;

The result is:

Title author publisher location

Construire application XML Jean Bernadac and François Knab Eyrolles Paris

XML en Action William J. Pardi Microsoft Press Paris


JSON Table Type

The Jpath Specification

Specification Array Type Description

[n] All Take the nth value of the array. Ignore it if n is 0.

[X] or [x] All Expand. Generate one row for each array value.

["string”] String Concatenate all values separated by the specified string.

[+] Numeric Make the sum of all the array values.

[*] Numeric Make the product of all array values.

[!] Numeric Make the average of all the array values.

[>] or [<] All Return the greatest or least value of the array.

[#] All Return the number of values in the array.

[] All Sum if numeric, else concatenation separated by “, “.

All Take the first value if an array.


Questions?

www.mariadb.com

www.facebook.com/mariadb.dbms

www.twitter.com/mariadb

------------------------------------

Serge Frezefond, Cloud Solution Architect

[email protected]

@sfrezefond


mailto:[email protected]


mariadb pres at lemug

Software