the native ndb engine for memcached
DESCRIPTION
From the O'Reilly MySQL Conference & Expo 2011. Integration between MySQL Cluster and memcached 1.6.TRANSCRIPT
![Page 1: The Native NDB Engine for Memcached](https://reader033.vdocument.in/reader033/viewer/2022050905/54b7bea74a79591c5d8b4578/html5/thumbnails/1.jpg)
1
![Page 3: The Native NDB Engine for Memcached](https://reader033.vdocument.in/reader033/viewer/2022050905/54b7bea74a79591c5d8b4578/html5/thumbnails/3.jpg)
<Insert Picture Here>
Program Agenda
• MySQL Cluster Today• A little bit about memcached• Design Decisions• Configuration• Performance• Links• Demo
3
![Page 4: The Native NDB Engine for Memcached](https://reader033.vdocument.in/reader033/viewer/2022050905/54b7bea74a79591c5d8b4578/html5/thumbnails/4.jpg)
The following is intended to outline our general product direction. It is intended for information purposes only, and may not be incorporated into any contract. It is not a commitment to deliver any material, code, or functionality, and should not be relied upon in making purchasing decisions.The development, release, and timing of any features or functionality described for Oracle’s products remains at the sole discretion of Oracle.
4
![Page 5: The Native NDB Engine for Memcached](https://reader033.vdocument.in/reader033/viewer/2022050905/54b7bea74a79591c5d8b4578/html5/thumbnails/5.jpg)
<Insert Picture Here>
MySQL Cluster Today
5
![Page 6: The Native NDB Engine for Memcached](https://reader033.vdocument.in/reader033/viewer/2022050905/54b7bea74a79591c5d8b4578/html5/thumbnails/6.jpg)
NDB Data Nodes
API Nodes
Node Group 1 Node Group 2
The Basics
6
![Page 7: The Native NDB Engine for Memcached](https://reader033.vdocument.in/reader033/viewer/2022050905/54b7bea74a79591c5d8b4578/html5/thumbnails/7.jpg)
NDB Data NodesNode Group 1 Node Group 2
The Basics – Data Storage
• High Read and Write Performance– Automatic data partitioning, multi-master, parallel execution• High Availability• Fast Failover– sub-second fault detection and reconfiguration• Scalable– Using commodity hardware
7
![Page 8: The Native NDB Engine for Memcached](https://reader033.vdocument.in/reader033/viewer/2022050905/54b7bea74a79591c5d8b4578/html5/thumbnails/8.jpg)
The Basics – Data Access
• High throughput– tens-to-hundreds of thousands of transactions per second• Low latency– sub-millisecond response• Multiple Acceess Methods– SQL and NoSQL
API Nodes
8
![Page 9: The Native NDB Engine for Memcached](https://reader033.vdocument.in/reader033/viewer/2022050905/54b7bea74a79591c5d8b4578/html5/thumbnails/9.jpg)
Recent History
• Cluster 6.3 (2008)– multi-master replication, key-distribution awareness
• Cluster 7.0 (2009)– Multi-threaded NDB nodes– Ability to add nodes online to a running cluster
• Cluster 7.1 (2010)– ClusterJ– MySQL Cluster Manager – ndbinfo (and then MEM support for Cluster)– Windows support
• Plus– Extensive improvements to BLOB performance, locking
behavior, node restart times, etc. in monthly releases
9
![Page 10: The Native NDB Engine for Memcached](https://reader033.vdocument.in/reader033/viewer/2022050905/54b7bea74a79591c5d8b4578/html5/thumbnails/10.jpg)
Employee findEmployee(long id) { Employee employee = session.find(Employee.class, id); return employee;}
ClusterJ Example
• Runs faster than JDBC• Almost as fast as the C++ NDB API• 3 lines of code
10
![Page 11: The Native NDB Engine for Memcached](https://reader033.vdocument.in/reader033/viewer/2022050905/54b7bea74a79591c5d8b4578/html5/thumbnails/11.jpg)
MySQL Cluster 7.2 Beta
• Push-Down Joins– Many “SPJ” (Select/Project/Join) operations can be executed
at the data nodes rather than the MySQL server– Long-term effort (presented at this conference last year)– 50x improvement for some queries
• MySQL privilege tables can be stored in cluster• Memcache API
11
![Page 12: The Native NDB Engine for Memcached](https://reader033.vdocument.in/reader033/viewer/2022050905/54b7bea74a79591c5d8b4578/html5/thumbnails/12.jpg)
<Insert Picture Here>
About memcache
• It’s a cache!• It’s not your data store!• If it fails, you get a cache miss!
12
![Page 13: The Native NDB Engine for Memcached](https://reader033.vdocument.in/reader033/viewer/2022050905/54b7bea74a79591c5d8b4578/html5/thumbnails/13.jpg)
Two levels of hashing
Memcache
httpd memcached
memcached
memcachedmemcache Key
PHP/Perl
friends:12389
hash key to find data
hash key to pick server
13
![Page 14: The Native NDB Engine for Memcached](https://reader033.vdocument.in/reader033/viewer/2022050905/54b7bea74a79591c5d8b4578/html5/thumbnails/14.jpg)
Cache hit
Memcache
httpd
memcachedPHP/Perl
friends:12389
hash key to find data
VALUE friends:12389 0 31\r\n101, 11009, 11150, 55881, 77798 \r\n
hash key to pick server
14
![Page 15: The Native NDB Engine for Memcached](https://reader033.vdocument.in/reader033/viewer/2022050905/54b7bea74a79591c5d8b4578/html5/thumbnails/15.jpg)
Cache miss (1): fetch from DB
Memcache
httpd
memcachedPHP/Perl hash key to find data
hash key to pick server
NOT FOUND
mysql
MySQL Slave
SELECT friend_id FROM user_friends WHERE user_id = ?
15
![Page 16: The Native NDB Engine for Memcached](https://reader033.vdocument.in/reader033/viewer/2022050905/54b7bea74a79591c5d8b4578/html5/thumbnails/16.jpg)
Cache miss (2): manage cache
Memcache
httpd
memcachedPHP/Perl hash key to find data
set friends:12389 31\r\n
101, 11009, 11150, 55881, 77798 \r\n
16
![Page 17: The Native NDB Engine for Memcached](https://reader033.vdocument.in/reader033/viewer/2022050905/54b7bea74a79591c5d8b4578/html5/thumbnails/17.jpg)
Data change (1): Write to DB
mysql
httpd
PHP/Perl
MySQLMaster
INSERT INTO user_friends (user_id, friend_id) VALUES ( 12389, 999101);
17
![Page 18: The Native NDB Engine for Memcached](https://reader033.vdocument.in/reader033/viewer/2022050905/54b7bea74a79591c5d8b4578/html5/thumbnails/18.jpg)
Data change (2): manage cache
mysql
httpd
PHP/Perl
MySQLMaster
memcached
delete friends:12389 \r\n
18
![Page 19: The Native NDB Engine for Memcached](https://reader033.vdocument.in/reader033/viewer/2022050905/54b7bea74a79591c5d8b4578/html5/thumbnails/19.jpg)
Expected Latency & Throughput
memcache
httpd
PHP/Perl memcached
MySQL Slave
1,000s of operations/sec.~ 2 ms round trip
10,000s of operations/sec.
~ 200 µs round trip
mysql
19
![Page 20: The Native NDB Engine for Memcached](https://reader033.vdocument.in/reader033/viewer/2022050905/54b7bea74a79591c5d8b4578/html5/thumbnails/20.jpg)
Cost per n active users
memcache
httpd
PHP/Perl $1
$10
mysql
20
![Page 21: The Native NDB Engine for Memcached](https://reader033.vdocument.in/reader033/viewer/2022050905/54b7bea74a79591c5d8b4578/html5/thumbnails/21.jpg)
A little bit more history
• Memcached 1.2 (2007)– The 2000-line Facebook patch
• UDP support • Vector I/O and other “details”
– adding up to 25% improved CPU efficiency• Multithreaded (for a small number of libevent threads)• Compare-And-Set operation (CAS)
• Memcached 1.4 (2009): – Binary Protocol– SASL Authentication
• Memcached 1.6 (upcoming)– Storage Engines– Logging modules– Windows platform support
21
![Page 22: The Native NDB Engine for Memcached](https://reader033.vdocument.in/reader033/viewer/2022050905/54b7bea74a79591c5d8b4578/html5/thumbnails/22.jpg)
<Insert Picture Here>
Design Decisions
22
![Page 23: The Native NDB Engine for Memcached](https://reader033.vdocument.in/reader033/viewer/2022050905/54b7bea74a79591c5d8b4578/html5/thumbnails/23.jpg)
Goals
• Access NDB data from memcache clients
– Memcached perspective: • NDB is a reliable, write-scalable, replicated data store
– MySQL Cluster perspective: • memcache is an easy-to-use high performance API
23
![Page 24: The Native NDB Engine for Memcached](https://reader033.vdocument.in/reader033/viewer/2022050905/54b7bea74a79591c5d8b4578/html5/thumbnails/24.jpg)
Goals
• Support existing schemas and all MySQL data types
• Cache NDB data inside memcached– with automatic cache management– and flexibility to fine-tune (or disable) the cache policies
• Support the whole memcache protocol
• Acheive superior performance– latency as expected from memcached– throughput as expected from memcached
24
![Page 25: The Native NDB Engine for Memcached](https://reader033.vdocument.in/reader033/viewer/2022050905/54b7bea74a79591c5d8b4578/html5/thumbnails/25.jpg)
Which codebase?
memcached.org libmemcached build our own
Text Protocol ✔
Binary Protocol ✔ ✔
TCP ✔ ✔
UDP ✔
Authentication ✔
✔
25
![Page 26: The Native NDB Engine for Memcached](https://reader033.vdocument.in/reader033/viewer/2022050905/54b7bea74a79591c5d8b4578/html5/thumbnails/26.jpg)
memcached
Server Architecture
memcached memcached
• Memcached separate from ndbd– Definitely!– “m:n” ratio of memcache servers
to data nodes
• Memcached inside ndbd – Maybe!– we know how to do it
– “embedded” API-to-TC channel
– lower latency– network round trip + tens of
microseconds– Not as flexible
26
![Page 27: The Native NDB Engine for Memcached](https://reader033.vdocument.in/reader033/viewer/2022050905/54b7bea74a79591c5d8b4578/html5/thumbnails/27.jpg)
Decisions
• Use memcached 1.6 tree• Isolate the NDB-specific code into an ndb_engine
– which currently exists outside the memcached source tree– and runs in an unmodified memcached server
• Build a standalone server binary• Possibly also build an ndbmtd-embedded server• Share code with InnoDB Memcache team
– on basic configuration and user-visible concepts– and on changes that would allow a memcache server module
to be loaded into either ndbmtd or mysqld
27
![Page 28: The Native NDB Engine for Memcached](https://reader033.vdocument.in/reader033/viewer/2022050905/54b7bea74a79591c5d8b4578/html5/thumbnails/28.jpg)
Other decisions
• Memcache INCR & DECR– Fully supported– Atomic operations – Performed at the data node
• Memcache CAS (version id check)– Fully Supported
• either at local cache or at database– CAS check is pushed down to the data node– Ideally an update that comes from a non-memcache API node
should invalidate the CAS• we still need to decide the best design for this
28
![Page 29: The Native NDB Engine for Memcached](https://reader033.vdocument.in/reader033/viewer/2022050905/54b7bea74a79591c5d8b4578/html5/thumbnails/29.jpg)
Limitations
• The size of stored values is limited to the NDB row size– currently just less than 8 KB – vs. 1 MB typical limit for memcached– due to the lack of BLOB support in the async NDB API
29
![Page 30: The Native NDB Engine for Memcached](https://reader033.vdocument.in/reader033/viewer/2022050905/54b7bea74a79591c5d8b4578/html5/thumbnails/30.jpg)
mysqld
Architecture Overview
Application Application
NDB Cluster
Data Node
Data Node
Data Node
Data Node
memcached
NDB EngineNDB Engine
Wide-areaReplication
binlog stream
InnoDB
Local Cache
30
![Page 31: The Native NDB Engine for Memcached](https://reader033.vdocument.in/reader033/viewer/2022050905/54b7bea74a79591c5d8b4578/html5/thumbnails/31.jpg)
<Insert Picture Here>
Configuring It
31
![Page 32: The Native NDB Engine for Memcached](https://reader033.vdocument.in/reader033/viewer/2022050905/54b7bea74a79591c5d8b4578/html5/thumbnails/32.jpg)
What’s Configurable
• Does it use local cache?• Does it use NDB? • What columns hold the keys?• What columns hold the data?• Are memcache commands like DELETE and FLUSH
allowed to delete records from the database?• You decide all of this ...• on a “per-key-prefix” basis.
32
![Page 33: The Native NDB Engine for Memcached](https://reader033.vdocument.in/reader033/viewer/2022050905/54b7bea74a79591c5d8b4578/html5/thumbnails/33.jpg)
A Key Prefix
user:1248
the prefix the database key
33
![Page 34: The Native NDB Engine for Memcached](https://reader033.vdocument.in/reader033/viewer/2022050905/54b7bea74a79591c5d8b4578/html5/thumbnails/34.jpg)
Fundamentals
• memcached command line specifies a connectstring for a primary cluster
• primary = “where the config is stored”• The NDB Engine reads the configuration tables from
the ndbmemcache database on the primary cluster• You, the administrator, manage the config via SQL• An NDB Memcache server can operate on data in the
primary cluster or in other clusters• Different NDB Memcache servers can fetch different
configurations
34
![Page 35: The Native NDB Engine for Memcached](https://reader033.vdocument.in/reader033/viewer/2022050905/54b7bea74a79591c5d8b4578/html5/thumbnails/35.jpg)
Standard Tables in ndbmemcache
• meta– stores configuration schema version (for upgrade
compatibility); consider it to be read-only
• ndb_clusters• containers
– where data is stored• cache_policies
– how it can be accessed
• key_prefixes• memcache_server_roles• last_memcached_signon• demo_table
35
![Page 36: The Native NDB Engine for Memcached](https://reader033.vdocument.in/reader033/viewer/2022050905/54b7bea74a79591c5d8b4578/html5/thumbnails/36.jpg)
ndb_clusters
• cluster_id– int, referenced by key_prefixes
• ndb_connectstring– varchar(128) – how to reach this cluster
• microsec_rtt– default 250; used for internal performance tuning
36
![Page 37: The Native NDB Engine for Memcached](https://reader033.vdocument.in/reader033/viewer/2022050905/54b7bea74a79591c5d8b4578/html5/thumbnails/37.jpg)
containers
• name• schema_name• table_name (the existing table where your data lives)• key_columns (comma-separated) • value_columns (comma-separated)• flags (either a constant number, or a column name)• increment_column (optional, for INCR/DECR)• cas_column (optional)• expire_time_column (optional)
37
![Page 38: The Native NDB Engine for Memcached](https://reader033.vdocument.in/reader033/viewer/2022050905/54b7bea74a79591c5d8b4578/html5/thumbnails/38.jpg)
cache_policies
• policy_name• get_policy
– enum (cache_only, ndb_only, caching, disabled) • set_policy
– enum (cache_only, ndb_only, caching, disabled) • delete_policy
– enum (cache_only, ndb_only, caching, disabled)
• flush_from_db– enum(false, true)
38
![Page 39: The Native NDB Engine for Memcached](https://reader033.vdocument.in/reader033/viewer/2022050905/54b7bea74a79591c5d8b4578/html5/thumbnails/39.jpg)
A key-prefix mapping
Memcache key
prefixCluster Container Cache
Policy
39
![Page 40: The Native NDB Engine for Memcached](https://reader033.vdocument.in/reader033/viewer/2022050905/54b7bea74a79591c5d8b4578/html5/thumbnails/40.jpg)
keyprefix Cluster Con-
tainerCache Policy
keyprefix Cluster Con-
tainerCache Policy
keyprefix Cluster Con-
tainerCache Policy
keyprefix Cluster Con-
tainerCache Policy
ServerRole
ID
A memcache server role
40
![Page 41: The Native NDB Engine for Memcached](https://reader033.vdocument.in/reader033/viewer/2022050905/54b7bea74a79591c5d8b4578/html5/thumbnails/41.jpg)
key_prefixes
• server_role_id• key_prefix• cluster_id• policy• container
41
![Page 42: The Native NDB Engine for Memcached](https://reader033.vdocument.in/reader033/viewer/2022050905/54b7bea74a79591c5d8b4578/html5/thumbnails/42.jpg)
demo_table
• Not really part of the configuration ...
• mkey – varchar(250) NOT NULL PRIMARY KEY
• math_value– bigint unsigned
• cas_value– bigint unsigned
• string_value– varchar(7500)
42
![Page 43: The Native NDB Engine for Memcached](https://reader033.vdocument.in/reader033/viewer/2022050905/54b7bea74a79591c5d8b4578/html5/thumbnails/43.jpg)
<Insert Picture Here>
Performance
43
![Page 44: The Native NDB Engine for Memcached](https://reader033.vdocument.in/reader033/viewer/2022050905/54b7bea74a79591c5d8b4578/html5/thumbnails/44.jpg)
Measured Latency
!sec
1000
500
ICMPping
MemcachedDefaultEngine
NDB1-NodeCluster
NDB2-NodeCluster
98
184
351 374
Read
Write
avg.
min.
95th
memcachetest -t 2 -M 7000 -c 25000
44
![Page 45: The Native NDB Engine for Memcached](https://reader033.vdocument.in/reader033/viewer/2022050905/54b7bea74a79591c5d8b4578/html5/thumbnails/45.jpg)
• This Presentation– http://en.oreilly.com/mysql2011/
• Memcached 1.6 development tree– git clone git://github.com/memcached/memcached– cd memcached– git checkout -t origin/engine-pu
• MySQL Cluster 7.2 with Memcached Engine– bzr clone lp:~mysql/mysql-server/mysql-cluster-7.2-labs-memcached– or look for it at labs.mysql.com
• Mailing List– [email protected]
Links
45
![Page 46: The Native NDB Engine for Memcached](https://reader033.vdocument.in/reader033/viewer/2022050905/54b7bea74a79591c5d8b4578/html5/thumbnails/46.jpg)
Demo
• What we’ll see:– Memcache Server Window
• start memcached “sandbox”– default role (NDB-only)– mc-only role (plain memcached)– ndb-caching role (NDB with local cache)
– Client Window• run memcapable
– MyQSL Window• examine data and configuration
46
![Page 47: The Native NDB Engine for Memcached](https://reader033.vdocument.in/reader033/viewer/2022050905/54b7bea74a79591c5d8b4578/html5/thumbnails/47.jpg)
47