[b14] a mysql replacement by colin charles
DESCRIPTION
TRANSCRIPT
MariaDB - a MySQL replacement
Colin Charles, Team MariaDB, SkySQL Ab [email protected] | http://mariadb.org/
http://bytebot.net/blog/ | @bytebot on Twitter DB Tech Showcase, Osaka, Japan
18 June 2014
whoami• Work on MariaDB at SkySQL Ab
• Merged with Monty Program Ab, makers of MariaDB
• Formerly MySQL AB (exit: Sun Microsystems)
• Past lives include Fedora Project (FESCO), OpenOffice.org
Who are you?
• Developer?
• Operator? (DBA, sysadmin)
• A bit of both?
Global Top 20 sites1.Google
2.Facebook
3.YouTube
4.Yahoo!
5.Baidu
6.Wikipedia
7.QQ
8.Taobao
9.Twitter
10.Live
11.LinkedIn
12.Sina
13.Amazon
14.hao123.com
15.google.co.in
16.blogspot
17.weibo.com
18.wordpress.com
19.360.cn
20.yandex.ru
5W1H is MariaDB• Drop-in compatible MySQL replacement
• Community developed, Foundation backed, feature enhanced, backwards compatible, GPLv2 licensed
• Steady stream of releases in 4 years 4 months: 5.1, 5.2, 5.3, 5.5, 10.0, MariaDB Galera Cluster 5.5, MariaDB with TokuDB 5.5
• Enterprise features open: PAM authentication plugin, threadpool, audit plugin
• Default in Red Hat Enterprise Linux, Fedora, openSUSE, etc.
MariaDB in Japan• Contributions:
• SPIDER storage engine
• Groonga full-text search engine
• HandlerSocket key/value store
• Translations
• 13.6% (6/44) maria-captains from Japan
• 24/7/365 support by Ashisuto
Microseconds• TIME, DATETIME, TIMESTAMP, temporal functions,
CAST, dynamic columns
CREATE TABLE microsec(
column_microsec DATETIME(6),
column_millisec TIME(3)
);
SELECT CURTIME(6);
MariaDB 5.3+
Microseconds & 5.6• TIME_TO_SEC(), UNIX_TIMESTAMP() preserve
microseconds of the argument
MariaDB 10.0 MySQL 5.6
SELECT TIME_TO_SEC('10:10:10.12345'); +-------------------------------+ | TIME_TO_SEC('10:10:10.12345') | +-------------------------------+ | 36610.12345 | +-------------------------------+ 1 row in set (0.01 sec)
SELECT TIME_TO_SEC('10:10:10.12345'); +-------------------------------+ | TIME_TO_SEC('10:10:10.12345') | +-------------------------------+ | 36610 | +-------------------------------+ 1 row in set (0.00 sec)
Virtual Columns• A column in a table that has its value automatically
calculated either with a pre-calculated/deterministic expression or values of other fields in the table
• VIRTUAL - computed on the fly when data is queried (like a VIEW)
• PERSISTENT - computed when data is inserted and stored in a table
MariaDB 5.2+
Virtual ColumnsCREATE TABLE table1 (
a INT NOT NULL,
b VARCHAR(32),
c INT AS (a mod 10) VIRTUAL,
d VARCHAR(5) AS (left(b,5)) PERSISTENT);
Virtual columns exampleCREATE TABLE product (
-> productname VARCHAR(25),
-> price_eur DOUBLE,
-> xrate DOUBLE,
-> price_cny DOUBLE AS (price_eur*xrate) VIRTUAL);
INSERT INTO product VALUES ('toothpaste', 1.5, 1.39, default);
INSERT into product VALUES ('shaving cream', 3.59, 1.39, default);
Virtual columns example IIselect * from product;
+---------------+-----------+-------+-------------------+
| productname | price_eur | xrate | price_cny |
+---------------+-----------+-------+-------------------+
| toothpaste | 1.5 | 1.39 | 2.085 |
| shaving cream | 3.59 | 1.39 | 4.990099999999999 |
+---------------+-----------+-------+-------------------+
2 rows in set (0.00 sec)
PCRE Regular Expressions• Powerful REGEXP/RLIKE operator
• New operators:
• REGEXP_REPLACE(sub,pattern,replace)
• REGEXP_INSTR(sub,pattern)
• REGEXP_SUBSTR(sub,pattern)
• Works with multi-byte character sets that MariaDB supports, including East-Asian sets
MariaDB 10.0+
GIS• MariaDB implements a subset of SQL with
Geometry Types
• No longer just minimum bounding rectangles (MBR) - shapes considered
CREATE TABLE geom (g GEOMETRY NOT NULL, SPATIAL INDEX(g)) ENGINE=MyISAM;
• ST_ prefix - as per OpenGIS requirements
MariaDB 5.3+
Dynamic columns• Allows you to create virtual columns with dynamic content for each
row in table. Store different attributes for each item (like a web store).
• Basically a BLOB with handling functions: COLUMN_CREATE, COLUMN_ADD, COLUMN_GET, COLUMN_DELETE, COLUMN_EXISTS, COLUMN_LIST, COLUMN_CHECK, COLUMN_JSON
• In MariaDB 10.0: name support (instead of referring to columns by numbers, name it), convert all dynamic column content to JSON array, interface with Cassandra
INSERT INTO tbl SET dyncol_blob=COLUMN_CREATE("column_name", "value");
MariaDB 5.3+
Query Cassandra• Data is mapped: rowkey, static columns, dynamic
columns
• super columns aren’t supported
• No 1-1 direct map for data types
• Write to Cassandra from SQL (SELECT, INSERT, UPDATE, DELETE)
MariaDB 10.0+
Cassandra II pk varchar(36) primary key,
data1 varchar(60),
data2 bigint
) engine=cassandra keyspace='ks1' column_family='cf1'
• Table must have a primary key
• name/type must match Cassandra’s rowkey
• Columns map to Cassandra’s static columns
• name must be same as in Cassandra, datatypes must match, can be subset of CF’s columns
Mapping
• Datatype mapping - complete table at KB
• Data mapping is safe - engine will refuse incorrect mappings
• Command mapping: INSERT overwrites rows, UPDATE reads then writes, DELETE reads then writes
Typical use cases
• Web page hits collection, streaming data
• Sensor data
• Reads served with a lookup
• Want an auto-replicated, fault-tolerant table?
CONNECT• Target: ETL for BI or analytics
• Import data from CSV, XML, ODBC, MS Access, etc.
• WHERE conditions pushed to ODBC source
• DROP TABLE just removes the stored definition, not data itself
• “Virtual” tables cannot be indexed
MariaDB 10.0+
SPIDER
• Horizontal partitioning, built on top of PARTITIONs
• Associates a partition with a remote server
• Transparent to user, easy to expand
• Has index condition pushdown support enabled
MariaDB 10.0+
TokuDB• Opensource - separate MariaDB 5.5+TokuDB/
integrated in 10.0.5
• Improved insert (10-20x faster) & query speed, compression (up to 90% space reduction), replication performance and online schema flexibility
• Uses Fractal Tree Indexes instead of B-Tree
• Tests & builds of TokuDB on multiple platforms
Engines, etc• Plan for backups - TokuDB can be cool for your uses as
an example
• Galera: study your workload patterns, your application, etc.
• SPIDER (built-in sharding capabilities, partitioning & XA transaction capable with multiple backends including Oracle)
• its not going to be straightforward to “just start” - need to know right tables to implement, etc.
Threadpool• Modified from 5.1 (libevent based), great for
CPU bound loads and short running queries
• Windows (threadpool), Linux (epoll), Solaris (event ports), FreeBSD/OSX (kevents)
• No minimization of concurrent transactions with dynamic pool size
• thread_handling=pool-of-threads
• https://mariadb.com/kb/en/thread-pool-in-mariadb-55/
MariaDB 5.5+
PAM Authentication• Authentication using /etc/shadow
• Authentication using LDAP, SSH pass phrases, password expiration, username mapping, logging every login attempt, etc.
• INSTALL PLUGIN pam SONAME ‘auth_pam.so’;
• CREATE USER foo@host IDENTIFIED via pam
• Remember to configure PAM (/etc/pam.d or /etc/pam.conf)
• http://www.mysqlperformanceblog.com/2013/02/24/using-two-factor-authentication-with-percona-server/
MariaDB 5.2+
SQL Error Logging Plugin
• Log errors sent to clients in a log file that can be analysed later. Log file can be rotated (recommended)
• a MYSQL_AUDIT_PLUGIN
install plugin SQL_ERROR_LOG soname 'sql_errlog.so';
MariaDB 5.5+
Audit Plugin
• Log server activity - who connects to the server, what queries run, what tables touched - rotating log file or syslogd
• a MYSQL_AUDIT_PLUGIN
INSTALL PLUGIN server_audit SONAME ‘server_audit.so’;
MariaDB 10.0+
Replication made better• Selective skipping of replication events (session-
based or on master or slave)
• Dynamic control of replication variables (no restarts!)
• Using row-based replication? Annotate the binary log with SQL statements
• Slaves perform checksums on binary log events
MariaDB 5.3+
Replication made better II• Group commit in the binary log - finally, sync_binlog=1, innodb_flush_log_at_trx_commit=1 performs
• START TRANSACTION WITH CONSISTENT SNAPSHOT
• mysqldump —single-transaction —master-data - full non-blocking backup
• Slaves crash-safe (data stored inside transaction tables)
• Multi-source replication - (real-time) analytics, shard provisioning, backups, etc.
New KILL syntax• HARD | SOFT & USER USERNAME are MariaDB-specific (5.3.2)
• KILL QUERY ID query_id (10.0.5) - kill by query id, rather than thread id
• SOFT ensures things that may leave a table in an inconsistent state aren’t interrupted (like REPAIR or INDEX creation for MyISAM or Aria)
KILL [HARD | SOFT] [CONNECTION | QUERY] [thread_id | USER user_name]
MariaDB 5.3+
Statistics• Understand server activity better to understand database loads
• SET GLOBAL userstat=1;
• SHOW CLIENT_STATISTICS; SHOW USER_STATISTICS;
• # of connections, CPU usage, bytes received/sent, row statistics
• SHOW INDEX_STATISTICS; SHOW TABLE_STATISTICS;
• # rows read, changed, indexes
• INFORMATION_SCHEMA.PROCESSLIST has MEMORY_USAGE, EXAMINED_ROWS (similar with SHOW STATUS output)
MariaDB 5.2+
MariaDB 10.0+
EXPLAIN enhanced• Explain analyser: https://mariadb.org/
explain_analyzer/analyze/
• SHOW EXPLAIN for <thread_id>
• EXPLAIN output in the slow query log
• EXPLAIN not just for SELECT but INSERT/UPDATE/DELETE
MariaDB 10.0+
Roles• Bundles users together, with similar privileges -
follows the SQL standard
CREATE ROLE audit_bean_counters;
GRANT SELECT ON accounts.* to audit_bean_counters;
GRANT audit_bean_counters to ceo;
MariaDB 10.0+
Connectors• The MariaDB project provides LGPL connectors
(client libraries) for:
• C
• Java
• ODBC
• Embedding a connector? Makes sense to use these LGPL licensed ones…
OptimizerMariaDB 10 MySQL 5.6
index_merge=on index_merge_union=on index_merge_sort_union=on index_merge_intersection=on index_merge_sort_intersection=off engine_condition_pushdown=off index_condition_pushdown=on derived_merge=on derived_with_keys=on firstmatch=on loosescan=on materialization=on in_to_exists=on semijoin=on partial_match_rowid_merge=on partial_match_table_scan=on subquery_cache=on mrr=off mrr_cost_based=off mrr_sort_keys=off outer_join_with_cache=on semijoin_with_cache=on join_cache_incremental=on join_cache_hashed=on join_cache_bka=on optimize_join_buffer_size=off table_elimination=on extended_keys=on exists_to_in=off
index_merge=on index_merge_union=on index_merge_sort_union=on index_merge_intersection=on engine_condition_pushdown=on index_condition_pushdown=on mrr=on mrr_cost_based=on block_nested_loop=on batched_key_access=off materialization=on semijoin=on loosescan=on firstmatch=on subquery_materialization_cost_based=on use_index_extensions=on
MariaDB 5.3+
MariaDB Galera Cluster• MariaDB Galera Cluster is made for today’s cloud
based environments. It is fully read-write scalable, comes with synchronous replication, allows multi-master topologies, and guarantees no lag or lost transactions.
• Currently 5.5-based
• 10.0 is in beta (almost ready for release)
Trusted by many• Google
• Wikipedia
• Tumblr
• SpamExperts
• Limelight Networks
• KakaoTalk
• Paybox Services
https://mariadb.com/kb/en/
Resources• We moved to github! https://github.com/MariaDB/server
• We’re still on launchpad for older branches: https://launchpad.net/maria
• #maria on freenode
• facebook.com/MariaDB.dbms
• @mariadb / +MariaDB
Q&[email protected] | [email protected] http://skysql.com/ | http://mariadb.org/
twitter: @bytebot | url: http://bytebot.net/blog/