handlersocket - a nosql plugin for mysql
DESCRIPTION
Presented on osdc.tw, 2011TRANSCRIPT
HANDLERSOCKET - A NOSQL PLUGIN FOR MYSQL
Jui-Nan Lin @OSDC.TW 2011
About this talk & me
URL of this Slide:
http://0rz.tw/jqQqY QRCode: About me:
Jui-Nan Lin (jnlin) Developer @
www.pixnet.net Twitter: @jnlin Github: jnlin
2
What is HandlerSocket?
“A NoSQL interface for MySQL” Only supports queries which use index Developed by DeNA (a Japanese social
gaming provider) https://github.com/ahiguti/HandlerSocket
-Plugin-for-MySQL
3
Architecture4
libmysql libhsclient
Application (Ex. PHP, Ruby, Perl …)
Listener for libmysql
SQL Layer
Handler Interface
InnoDB MyISAM Other storage engines
HandlerSocket Plugin
application
mysqld
Reference: http://www.slideshare.net/akirahiguchi/handlersocket-plugin-for-mysql-4664154
NoSQL for Web Application
Most operations in web application are simple CRUD.
RDBMS is too powerful for web application.
Typical solution: Memcached + MySQL Cassandra Apache CouchDB Amazon SimpleDB
5
NoSQL for Web Application
Pros: Scalable Fast
Cons: Unstable
Twitter drops Cassandra in 2010: http://engineering.twitter.com/2010/07/cassandra-at-twitter-today.html
6
HandlerSocket for MySQL
MySQL is very stable HandlerSocket provides a NoSQL
interface for MySQL HandlerSocket is very fast. Faster than
MySQL.
7
How fast is it
750,000 qps on a commodity server (tested by original developers) MySQL 5.1 + E5540 2.53Ghz + 32 GB RAM All data fits in memory All Clients are written in C 100,000 qps with Original MySQL
About 7.5x faster than original MySQL!
8
How fast is it
1m PK SELECT queries each test on 1 table contains 1m rows. Client is written in PHP. SELECT * from table1 WHERE pk = ?
9
Concurrent Connections
1 10 50 100
HandlerSocket
Time (sec)
256 28 15 13
QPS 3906 35714 66667 76923
MySQL
Time (sec)
372 83 87 85
QPS 2688 12048 11494 11765
Why HandlerSocket is fast10
from http://yoshinorimatsunobu.blogspot.com/2010/10/using-mysql-as-nosql-story-for.html
Why HandlerSocket is fast
HandlerSocket do not need to do: Parsing SQL Statements Opening, locking table Making SQL execution plans
Index usage Unlocking, closing tables
11
Why HandlerSocket is fast
HandlerSocket does do: Plain text-based protocol Keep table open for reuse Event based (use epoll)
Many Connection per MySQL thread Accept only index queries
Support range query, LIMIT, multi_get operations
12
Language Bindings
PHP: php-handlersocket Java: hs4j Python: python-handler-socket Ruby: ruby-handlersocket Perl: Net::HanderSocket
13
Programming Example
PHP Read Operation Port used for write operation is different
14
Configuration (my.cnf)
handlersocket_port = 9998 Port for reading
handlersocket_port_wr = 9999 Port for writing
handlersocket_threads = 16 No. of reader threads Recommended Value: # of CPUs
handlersocket_threads_wr = 1 No. of writer threads
open_files_limit = 65535
15
Pros and Cons
Pros Stable High Performance Support a lot of query types No duplicate cache (compared with MySQL
+ memcached) Invalidation Problem
Flexible Support SQL interface. We can still use SQL
statements! (ex. report generating) Write binlog and support MySQL replication.
Included in Percona Server
16
Pros and Cons
Cons No Security
Without username and password Do not update MySQL Query Cache
Fixed in recent commits No “INSERT … ON DUPLICATED KEY
UPDATE …” Feature Do not support auto increment column
while inserting Fixed in recent commits
17
Q&A
Thank you!
18