mysql binary log api presentation - oscon 2011
DESCRIPTION
Presentation of the new MySQL binary log API from OSCON in Portland.TRANSCRIPT
![Page 1: MySQL Binary Log API Presentation - OSCON 2011](https://reader034.vdocument.in/reader034/viewer/2022051412/54c674504a79593d1c8b457a/html5/thumbnails/1.jpg)
![Page 2: MySQL Binary Log API Presentation - OSCON 2011](https://reader034.vdocument.in/reader034/viewer/2022051412/54c674504a79593d1c8b457a/html5/thumbnails/2.jpg)
<Insert Picture Here>
Lars ThalmannDevelopment Director, MySQLReplication, Backup & Connectors
An API for Reading the MySQL Binary Log
Mats KindahlLead Software Engineer, MySQLReplication & Utilities
![Page 3: MySQL Binary Log API Presentation - OSCON 2011](https://reader034.vdocument.in/reader034/viewer/2022051412/54c674504a79593d1c8b457a/html5/thumbnails/3.jpg)
3
The following is intended to outline our general product direction. It is intended for information purposes only, and may not be incorporated into any contract. It is not a commitment to deliver any material, code, or functionality, and should not be relied upon in making purchasing decisions.The development, release, and timing of any features or functionality described for Oracle’s products remains at the sole discretion of Oracle.
![Page 4: MySQL Binary Log API Presentation - OSCON 2011](https://reader034.vdocument.in/reader034/viewer/2022051412/54c674504a79593d1c8b457a/html5/thumbnails/4.jpg)
4
Outline
•Replication Architecture
•Binary logs
•Binary log event
•Reading binary log– Connecting to server– Reading from files
•Reading events– Queries– Reading rows (row-based replication)– Other events
![Page 5: MySQL Binary Log API Presentation - OSCON 2011](https://reader034.vdocument.in/reader034/viewer/2022051412/54c674504a79593d1c8b457a/html5/thumbnails/5.jpg)
5
Replication Architecture
Master Slave(s)
Clients
Changes
![Page 6: MySQL Binary Log API Presentation - OSCON 2011](https://reader034.vdocument.in/reader034/viewer/2022051412/54c674504a79593d1c8b457a/html5/thumbnails/6.jpg)
6
I/O SQL
Database
Replication Architecture
ClientDumpDumpDump
Binary Log
Session
Session
SessionI/O SQL
Database
I/O SQL
Database
Master Slave(s)
![Page 7: MySQL Binary Log API Presentation - OSCON 2011](https://reader034.vdocument.in/reader034/viewer/2022051412/54c674504a79593d1c8b457a/html5/thumbnails/7.jpg)
7
I/O SQL
Database
Replication Architecture
ClientDumpDumpDump
Binary Log Relay Log
Session
Session
SessionI/O SQL
Database
I/O SQL
Database
Master Slave(s)
![Page 8: MySQL Binary Log API Presentation - OSCON 2011](https://reader034.vdocument.in/reader034/viewer/2022051412/54c674504a79593d1c8b457a/html5/thumbnails/8.jpg)
8
Replication to other systems
Client
Binary Log
Dump
Session
Session
Session
Master
Relay Log
I/O SQL
Database
Slave(s)
HBase
SOLR Full-text indexing
Data Mining
?
DumpDumpDump
![Page 9: MySQL Binary Log API Presentation - OSCON 2011](https://reader034.vdocument.in/reader034/viewer/2022051412/54c674504a79593d1c8b457a/html5/thumbnails/9.jpg)
9
Transforming events
SOLRAPI Transformer
Server
File
Subject of our presentation
“Change Data Capture”
![Page 10: MySQL Binary Log API Presentation - OSCON 2011](https://reader034.vdocument.in/reader034/viewer/2022051412/54c674504a79593d1c8b457a/html5/thumbnails/10.jpg)
10
Binlog API
•Library to process replication events
•API is ready for use
•Goals:
–Simple–Extensible–Efficient
![Page 11: MySQL Binary Log API Presentation - OSCON 2011](https://reader034.vdocument.in/reader034/viewer/2022051412/54c674504a79593d1c8b457a/html5/thumbnails/11.jpg)
11
<Insert Picture Here>
How to capture events
Binlog API●The replication listener
![Page 12: MySQL Binary Log API Presentation - OSCON 2011](https://reader034.vdocument.in/reader034/viewer/2022051412/54c674504a79593d1c8b457a/html5/thumbnails/12.jpg)
12
<Insert Picture Here>
First example#include <cstdlib>#include <iostream>#include <binlog_api.h>
int main(int argc, char *argv[]) { const char *url = “mysql://[email protected]:3360”; Binary_log binlog(create_transport(url)); binlog.connect(); Binary_log_event *event; while (true) { int result = binlog.wait_for_next_event(&event); if (result == ERR_EOF) break; cout << “ at “ << binlog.get_position() << “ event type “ << event.get_type_code() << endl; } return EXIT_SUCCESS;}
![Page 13: MySQL Binary Log API Presentation - OSCON 2011](https://reader034.vdocument.in/reader034/viewer/2022051412/54c674504a79593d1c8b457a/html5/thumbnails/13.jpg)
13
<Insert Picture Here>
Create network transport#include <cstdlib>#include <iostream>#include <binlog_api.h>
int main(int argc, char *argv[]) { const char *url = “mysql://[email protected]:3360”; Binary_log binlog(create_transport(url)); binlog.connect(); Binary_log_event *event; while (true) { int result = binlog.wait_for_next_event(&event); if (result == ERR_EOF) break; cout << “ at “ << binlog.get_position() << “ event type “ << event.get_type_code() << endl; } return EXIT_SUCCESS;}
![Page 14: MySQL Binary Log API Presentation - OSCON 2011](https://reader034.vdocument.in/reader034/viewer/2022051412/54c674504a79593d1c8b457a/html5/thumbnails/14.jpg)
14
<Insert Picture Here>
… or file transport#include <cstdlib>#include <iostream>#include <binlog_api.h>
int main(int argc, char *argv[]) { const char *url = “file:///tmp/binlog.0000001”; Binary_log binlog(create_transport(url)); binlog.connect(); Binary_log_event *event; while (true) { int result = binlog.wait_for_next_event(&event); if (result == ERR_EOF) break; cout << “ at “ << binlog.get_position() << “ event type “ << event.get_type_code() << endl; } return EXIT_SUCCESS;}
![Page 15: MySQL Binary Log API Presentation - OSCON 2011](https://reader034.vdocument.in/reader034/viewer/2022051412/54c674504a79593d1c8b457a/html5/thumbnails/15.jpg)
15
<Insert Picture Here>
Connect the transport#include <cstdlib>#include <iostream>#include <binlog_api.h>
int main(int argc, char *argv[]) { const char *url = “file:///tmp/binlog.0000001”; Binary_log binlog(create_transport(url)); binlog.connect(); Binary_log_event *event; while (true) { int result = binlog.wait_for_next_event(&event); if (result == ERR_EOF) break; cout << “ at “ << binlog.get_position() << “ event type “ << event.get_type_code() << endl; } return EXIT_SUCCESS;}
![Page 16: MySQL Binary Log API Presentation - OSCON 2011](https://reader034.vdocument.in/reader034/viewer/2022051412/54c674504a79593d1c8b457a/html5/thumbnails/16.jpg)
16
<Insert Picture Here>
Digression: set read position
•Default: start at beginning
•Set position explicitly:if (binlog.set_position(file, pos)){ /* Handle error */}
![Page 17: MySQL Binary Log API Presentation - OSCON 2011](https://reader034.vdocument.in/reader034/viewer/2022051412/54c674504a79593d1c8b457a/html5/thumbnails/17.jpg)
17
<Insert Picture Here>
Read events#include <cstdlib>#include <iostream>#include <binlog_api.h>
int main(int argc, char *argv[]) { const char *url = “file:///tmp/binlog.0000001”; Binary_log binlog(create_transport(url)); binlog.connect(); Binary_log_event *event; while (true) { int result = binlog.wait_for_next_event(&event); if (result == ERR_EOF) break; cout << “ at “ << binlog.get_position() << “ event type “ << event>get_type_code() << endl; } return EXIT_SUCCESS;}
Get event
![Page 18: MySQL Binary Log API Presentation - OSCON 2011](https://reader034.vdocument.in/reader034/viewer/2022051412/54c674504a79593d1c8b457a/html5/thumbnails/18.jpg)
18
Steps summary
•Create a transport– create_transport
•Connect to server– connect
•Set position– set_position
•Start event loop– wait_for_next_event
![Page 19: MySQL Binary Log API Presentation - OSCON 2011](https://reader034.vdocument.in/reader034/viewer/2022051412/54c674504a79593d1c8b457a/html5/thumbnails/19.jpg)
19
<Insert Picture Here>
Reading information in events
Binlog API●The replication listener
![Page 20: MySQL Binary Log API Presentation - OSCON 2011](https://reader034.vdocument.in/reader034/viewer/2022051412/54c674504a79593d1c8b457a/html5/thumbnails/20.jpg)
20
Binlog Event Structure
Post-header
Variable Part
Common Header
• Common header• Generic data• Fixed size
• Post-header• Event-specific data• Fixed size
• Variable part• Event-specific data• Variable size
![Page 21: MySQL Binary Log API Presentation - OSCON 2011](https://reader034.vdocument.in/reader034/viewer/2022051412/54c674504a79593d1c8b457a/html5/thumbnails/21.jpg)
21
<Insert Picture Here>
Reading the header
•Read common header– header()
•Access fields
switch (event>header()>type_code) {case QUERY_EVENT: …case USER_VAR_EVENT: …case FORMAT_DESCRIPTION_EVENT: …}
![Page 22: MySQL Binary Log API Presentation - OSCON 2011](https://reader034.vdocument.in/reader034/viewer/2022051412/54c674504a79593d1c8b457a/html5/thumbnails/22.jpg)
22
Binlog Event Common Header
19 Bytes
4 bytes
timestamptype_code
server_id
event_lengthflags
next_position
• Data common to all events
• Next Position– One-after-end of event
• Timestamp– Statement start time
• Flags– Binlog-in-use
– Thread-specific
– Suppress “use”
– Artificial
– Relay-log event
![Page 23: MySQL Binary Log API Presentation - OSCON 2011](https://reader034.vdocument.in/reader034/viewer/2022051412/54c674504a79593d1c8b457a/html5/thumbnails/23.jpg)
23
Binlog Event Structure
Post-header
Variable Part
Common Header
• Common header• Generic data• Fixed size
• Post-header• Event-specific data• Fixed size
• Variable part• Event-specific data• Variable size
![Page 24: MySQL Binary Log API Presentation - OSCON 2011](https://reader034.vdocument.in/reader034/viewer/2022051412/54c674504a79593d1c8b457a/html5/thumbnails/24.jpg)
24
Query Event
•Most common event
•Used for statements
•Statement logged literally– … in almost all casesquery
Common Header
thread_id exec_time
error_code
std::vector<uint8_t> variables
Special case
: need to be
decoded
db_name
![Page 25: MySQL Binary Log API Presentation - OSCON 2011](https://reader034.vdocument.in/reader034/viewer/2022051412/54c674504a79593d1c8b457a/html5/thumbnails/25.jpg)
25
<Insert Picture Here>
Reading event data
•Cast to correct event type
•Access fields
switch (event>header()>type_code) {case QUERY_EVENT: Query_event *qev = static_cast<Query_event*>(event); cout << qev>query << endl; break;case USER_VAR_EVENT: …case FORMAT_DESCRIPTION_EVENT: …}
![Page 26: MySQL Binary Log API Presentation - OSCON 2011](https://reader034.vdocument.in/reader034/viewer/2022051412/54c674504a79593d1c8b457a/html5/thumbnails/26.jpg)
26
<Insert Picture Here>
Event-driven API
![Page 27: MySQL Binary Log API Presentation - OSCON 2011](https://reader034.vdocument.in/reader034/viewer/2022051412/54c674504a79593d1c8b457a/html5/thumbnails/27.jpg)
27
<Insert Picture Here>
Event-driven API
• Content handlers
wait_for_next_event
![Page 28: MySQL Binary Log API Presentation - OSCON 2011](https://reader034.vdocument.in/reader034/viewer/2022051412/54c674504a79593d1c8b457a/html5/thumbnails/28.jpg)
28
Saving user-defined variables
class Save_handler : public Content_handler{ … };
Save_handler::Map vars;Save_handler save_vars(vars);binlog.content_handler_pipeline() >push_back(&save_vars);
![Page 29: MySQL Binary Log API Presentation - OSCON 2011](https://reader034.vdocument.in/reader034/viewer/2022051412/54c674504a79593d1c8b457a/html5/thumbnails/29.jpg)
29
<Insert Picture Here>
User-defined variablesclass Save_handler : public Content_handler {public: typedef std::map<std::string, std::string> Map; Save_handler(Map &container) : m_var(container) { }
Binary_log_event * process_event(User_var_event *event) { m_var[event>name] = event>value; return NULL; }
private: Map &m_var;};
![Page 30: MySQL Binary Log API Presentation - OSCON 2011](https://reader034.vdocument.in/reader034/viewer/2022051412/54c674504a79593d1c8b457a/html5/thumbnails/30.jpg)
30
Replace handler
class Replace_vars : public Content_handler{ Binary_log_event * process_event(Query_log_event *event) { /* Code to replace variables */ }};
Full ex
ample:
basic-
2.cpp
![Page 31: MySQL Binary Log API Presentation - OSCON 2011](https://reader034.vdocument.in/reader034/viewer/2022051412/54c674504a79593d1c8b457a/html5/thumbnails/31.jpg)
31
<Insert Picture Here>
Example two:
How to capture live row changes
Binlog API●The replication listener
![Page 32: MySQL Binary Log API Presentation - OSCON 2011](https://reader034.vdocument.in/reader034/viewer/2022051412/54c674504a79593d1c8b457a/html5/thumbnails/32.jpg)
32
Row events in the binlog
Table map
Write rows
Write rows
Write rows
Table map
Delete rows
Transaction
Transaction
Header
Map table definitionto table ID
We'll cover thissoon (trust me)
A bunch of rows
![Page 33: MySQL Binary Log API Presentation - OSCON 2011](https://reader034.vdocument.in/reader034/viewer/2022051412/54c674504a79593d1c8b457a/html5/thumbnails/33.jpg)
33
Capturing row events
class Row_event_handler : public Content_handler{public: Binary_log_event * process_event(Row_event *event) { switch(ev>header()>type_code) { case WRITE_ROWS_EVENT: case UPDATE_ROWS_EVENT: case DELETE_ROWS_EVENT: ...
![Page 34: MySQL Binary Log API Presentation - OSCON 2011](https://reader034.vdocument.in/reader034/viewer/2022051412/54c674504a79593d1c8b457a/html5/thumbnails/34.jpg)
34
Capturing row events
• The *_ROWS_EVENT
uint64_t table_id;uint16_t flags;uint64_t columns_len;uint32_t null_bits_len;vector<uint8_t> columns_before_image;vector<uint8_t> used_columns;vector<uint8_t> row;
Raw row data
Defined in thetable map event
![Page 35: MySQL Binary Log API Presentation - OSCON 2011](https://reader034.vdocument.in/reader034/viewer/2022051412/54c674504a79593d1c8b457a/html5/thumbnails/35.jpg)
35
Reading rows
•Wrap raw row data in Row_event_set
•Iterate over rows using iteratorRow_event_set rows(row_event, table_map_event);
Row_event_set::iterator it= rows.begin();
You need to havecaptured this before!
![Page 36: MySQL Binary Log API Presentation - OSCON 2011](https://reader034.vdocument.in/reader034/viewer/2022051412/54c674504a79593d1c8b457a/html5/thumbnails/36.jpg)
36
Reading fields of a row
•Row_of_fields to iterate fields of a row
–Turns row into row of fields sequenceRow_event_set rows(row_event, table_map_event);
for (Row_event_set::iterator it = rows.begin() ; it != rows.end() ; ++it) table_delete(os.str(), Row_of_fields(*it));
![Page 37: MySQL Binary Log API Presentation - OSCON 2011](https://reader034.vdocument.in/reader034/viewer/2022051412/54c674504a79593d1c8b457a/html5/thumbnails/37.jpg)
37
Reading fields of a row
• Iterate over fields in Row_of_fields
void table_delete (..., const Row_of_fields& fields){ Row_of_fields::iterator it= fields.begin(); for (int id = 0 ; it =! fields.end() ; ++it, ++id) { std::string str; Converter().to(str, *it); std::cout << id << "= " << str << std::endl; }}
![Page 38: MySQL Binary Log API Presentation - OSCON 2011](https://reader034.vdocument.in/reader034/viewer/2022051412/54c674504a79593d1c8b457a/html5/thumbnails/38.jpg)
38
Decoding a field
• Iterate over fields in Row_of_fields
void table_delete (..., const Row_of_fields& fields){ Row_of_fields::iterator it= fields.begin(); for (int id = 0 ; it =! fields.end() ; ++it, ++id) { std::string str; Converter().to(str, *it); std::cout << id << "= " << str << std::endl; }}
![Page 39: MySQL Binary Log API Presentation - OSCON 2011](https://reader034.vdocument.in/reader034/viewer/2022051412/54c674504a79593d1c8b457a/html5/thumbnails/39.jpg)
39
Summary – what's it for?
•Replicate to other systems– Hbase, SOLR, etc.
•Triggering on specific events– Call the DBA when tables are dropped?– Monitor objects in database
•Browsing binary logs– Extract subset of changes (by table, by execution time, etc.)– Statistics
•Component in building other solutions– Point-in-time restore– Sharding / Load-balancing
![Page 40: MySQL Binary Log API Presentation - OSCON 2011](https://reader034.vdocument.in/reader034/viewer/2022051412/54c674504a79593d1c8b457a/html5/thumbnails/40.jpg)
40
Summary – what we've covered
•Reading events
•Creating content handlers
•Processing queries
•Processing rows
•Reading fields
•… but there is a lot more
![Page 41: MySQL Binary Log API Presentation - OSCON 2011](https://reader034.vdocument.in/reader034/viewer/2022051412/54c674504a79593d1c8b457a/html5/thumbnails/41.jpg)
41
• Available at labs
– http://labs.mysql.com/• Source code available at launchpad
– http://launchpad.net/mysql-replication-listener• MySQL High Availability
Get it as free ebook: http://oreilly.com/go/ebookrequest Valid this week, mention event “MySQL Replication Update”
![Page 42: MySQL Binary Log API Presentation - OSCON 2011](https://reader034.vdocument.in/reader034/viewer/2022051412/54c674504a79593d1c8b457a/html5/thumbnails/42.jpg)