bucardo

endpoint.com

Bucardo:Replication with tiny little goats

http://www.flickr.com/photos/kevincollins/

Selena Deckelmannselena@endpoint.com

endpoint.com

★ Software Engineer / Postgres Leadhttp://endpoint.com

★ We’re hiring! http://tr.im/EMGj

Hi!

endpoint.com

http://www.telegraph.co.uk/science/science-news/4409958/Extinct-ibex-is-resurrected-by-cloning.html

endpoint.com

What can it do?

http://www.flickr.com/photos/8602783@N06/2294028540

endpoint.com

Master-slave

• Scaling/Redundancy

• UPGRADES (8.x -> 8.4)

• Reporting databases

• Data warehousing

• Now handles SEQUENCES!

endpoint.com

Master-master

• Write between both databases!

• Sync type: SWAP

• Conflict resolution

endpoint.com

Custom code

• Respond to change

• Filter

• Run arbitrary code!

endpoint.com

STORY TIME!

item onhandskis 10

itemskiitemskiitem

skiitemskiitem

skiitemskiitem

skiitemskiitem

skiitemski

on hand = count(item)

endpoint.com

itemski

new!

bucardo

count(items)and other stuff

prod dbreporting db

itemski

on hand+1

new!

custom code

NOTIFY

NO

TIFY

sync

A

sync Bsync C

goat courtesy of: http://www.flickr.com/photos/nuritwilde/394603312

endpoint.com

Strengths

• Drop-in, no changes to Postgres

• Control daemon/DB can run anywhere

• EXTREME async

• No locking of slaves

• Command-line controls rock!

endpoint.com

The Fine Print

• No smooth DDL handling

• Not for failover (aka high availability)

• bloat: pg_listener, q, bucardo_delta, bucardo_track

• No locking of slaves

• No Windows support

• One developer

http://www.flickr.com/photos/ncarey/135901120/

endpoint.comhttp://wiki.postgresql.org/wiki/Replication,_Clustering,_and_Connection_Pooling

endpoint.com

BucardoConcepts

http://www.flickr.com/photos/kendrick/361173694

endpoint.com

Glossary of terms

• replication: copying data from one place to another

• DDL: Data Definition Language (CREATE/DROP/ALTER)

• DML: Data Manipulation Language (SELECT/INSERT/UPDATE/DELETE)

• triggers: bits of code run before or after DML

endpoint.com

Glossary of terms

• goat: a database object (table or sequence)

• herd: table contains unique name for a group of goats

• syncs: replication events (one herd per sync)

endpoint.com

flexible architecture

bucardo control database

master DB slave DB

endpoint.com

flexible architecture

bucardo control database

master DB slave DB

same db cluster

endpoint.com

flexible architecture

bucardo control database

master DB slave DB

same db cluster

endpoint.com

flexible architecture

bucardo control database

slave DBmaster DB

same db cluster

endpoint.com

Installing

http://www.flickr.com/photos/kendrick/361173694

endpoint.com

Bucardo Requirements

• Postgres (8.3 or higher recommended)

• Perl & some packages:

• DBD::Pg 2.0

• DBI 1.51

• DBIx::Safe 1.2.4

• ExtUtils::MakeMaker 6.32

• pl/perlu and pl/pgsql

endpoint.com

Simple set up

BUCARDO CONTROL database setup:bucardo_ctl install

endpoint.com

Simple set up

MASTER setup:bucardo_ctl add db MyDB name=masterbucardo_ctl add all tables herd=all_tablesbucardo_ctl add all sequences herd=all_tables

SLAVE setup:bucardo_ctl add db MyDB name=slave port=6543

endpoint.com

Setting up, cont.

TEST:

bucardo_ctl validate all

REPLICATE:

bucardo_ctl add sync delta type=pushdelta source=all_tables targetdb=slavebucardo_ctl start

(installs triggers!)

endpoint.com

Setting up

• Initiate a one-time-copy: onetimecopy=[1|2]

• Set up ongoing syncs: enabled by default (can set timeouts)

• MANY configuration options

endpoint.com

Admin stuff

• Can set up defaults in .bucardorc

• Only need one ‘bucardo’ database

• Uses the ‘bucardo’ schema in replication sets for ease of admin, removal

endpoint.com

Smooth upgrades!

bucardo_ctl upgrade

• Stop bucardo first (bucardo_ctl stop)

• applies DDL changes, functions

• see also UPGRADE file

endpoint.com

Stuff to watch out for

• search_path for ‘bucardo’ user

• PL/Perlu & pl/pgsql must be installed

• Uses prepared statements, turn off with:

bucardo_ctl update db MyDB server_side_prepares=0

endpoint.com

LIVE DEMO!

• http://github.com/selenamarie/bucardo_examples

endpoint.com

Internals

http://www.flickr.com/photos/chelseagirlphotos

endpoint.com

Internalsbucardo=# \d List of relations Schema | Name | Type | Owner ---------+-------------------------------+----------+--------- bucardo | audit_pid | table | bucardo bucardo | audit_pid_id_seq | sequence | bucardo bucardo | bucardo_config | table | bucardo bucardo | bucardo_custom_trigger | table | bucardo bucardo | bucardo_custom_trigger_id_seq | sequence | bucardo bucardo | bucardo_log_message | table | bucardo bucardo | bucardo_rate | table | bucardo bucardo | customcode | table | bucardo bucardo | customcode_id_seq | sequence | bucardo bucardo | customcode_map | table | bucardo bucardo | db | table | bucardo bucardo | db_connlog | table | bucardo bucardo | dbgroup | table | bucardo bucardo | dbmap | table | bucardo bucardo | goat | table | bucardo bucardo | goat_id_seq | sequence | bucardo bucardo | herd | table | bucardo bucardo | herdmap | table | bucardo bucardo | q | table | bucardo bucardo | sync | table | bucardo bucardo | upgrade_log | table | bucardo(21 rows)

endpoint.com

goatsTable "bucardo.goat" Column | Type |----------------------+--------------------------+ id | integer | db | text | schemaname | text | tablename | text | reltype | text | pkey | text | qpkey | text | pkeytype | text | has_delta | boolean | ping | boolean | ...

endpoint.com

goats

# bucardo_ctl list tablesTable: archive.course_logging DB: odw_master PK: noneTable: archive.messages DB: odw_master PK: noneTable: archive.student_assignments DB: odw_master PK: noneTable: archive.student_courses DB: odw_master PK: noneTable: logging.activity DB: odw_master PK: activity_name (varchar)Table: logging.course_logging DB: odw_master PK: course_logging_id (int4)Table: public.ecash_transaction DB: odw_master PK: trans_id (int4)Table: public.feedback DB: odw_master PK: date (timestamp)Table: public.holidays DB: odw_master PK: customernum|month|day|year (int4|int4|int4|int4)...

endpoint.com

bucardo_ctl list sequencesSequence 63: logging.logging_seqSequence 64: public.trans_id_seqSequence 65: public.num_seqSequence 66: public.upload_seq

endpoint.com

bucardo databasebucardo=# \d List of relations Schema | Name | Type | Owner ---------+-------------------------------+----------+---------... bucardo | db | table | bucardo bucardo | db_connlog | table | bucardo bucardo | dbgroup | table | bucardo bucardo | dbmap | table | bucardo bucardo | goat | table | bucardo bucardo | goat_id_seq | sequence | bucardo bucardo | herd | table | bucardo bucardo | herdmap | table | bucardo bucardo | q | table | bucardo bucardo | sync | table | bucardo bucardo | upgrade_log | table | bucardo(21 rows)

endpoint.com

Status

# bucardo_ctl statusDays back: 3 User: bucardo Database: bucardo Port: 8000 PID of Bucardo MCP: 6876Name Type State PID Last_good Time I/U/D Last_bad Time========+=====+=====+====+=========+=====+=====+========+====a_delta | P |idle |6886|1m4s |0s |0/0/0|unknown | b_delta | P |idle |6885|1m4s |0s |0/0/0|9h32m47s|0s c_delta | P |idle |6887|1m4s |0s |0/0/0|unknown | d_delta | P |idle |6888|54s |0s |0/0/0|2m6s |3s e_delta | P |idle |6890|1m4s |0s |0/0/0|9h32m47s|0s f_delta | P |idle |6889|1m4s |0s |0/0/0|9h32m47s|0s

endpoint.com

# bucardo_ctl status a_deltaDays back: 3 User: bucardo Database: bucardo Port: 8000======================================================================Sync name: a_deltaCurrent state: idle (PID = 6888)Type: pushdeltaSource herd/database: all_a / a_masterTarget database: a_slaveTables in sync: 26Last good: 3m 34s (time to run: 0s)Last good time: Oct 16, 2009 17:52:20 Target: a_slaveIns/Upd/Del: 20 / 0 / 50Last bad: 4m 47s (time to run: 3s)Last bad time: Oct 16, 2009 17:51:08 Target: a_slaveLatest bad reason: ?PID file: /var/run/bucardo/bucardo.ctl.sync.a_delta.pidPID file created: Fri Oct 16 17:52:09 2009Status: activeLimitdbs: 0Priority: 0Checktime: noneOverdue time: 00:00:00Expired time: 00:00:00Stayalive: yes Kidsalive: yesRebuild index: 0 Do_listen: yesPing: yes Makedelta: no Onetimecopy: 0

endpoint.com

How Bucardo replicates

endpoint.com

bucardo_add_delta

bucardo_triggerkick_[sync]

bucardo_delta

INSERT

NOTIFY!

Triggers IN

SERT

, UPD

ATE,

DEL

ETE

endpoint.com

NOTIFY / LISTEN

• Simple interprocess communication

psql# LISTEN scotch;psql# NOTIFY scotch;Asynchronous notification "scotch" received from server process with PID 1337.

• Any Postgres process can listen in

endpoint.com

Actual Triggers! Table "public.test" Column | Type | Modifiers --------+-----------------------+----------- test | character varying(15) | not nullIndexes: "test_pkey" PRIMARY KEY, btree ("test")Triggers: bucardo_add_delta AFTER INSERT OR DELETE OR UPDATE ON test FOR EACH ROW EXECUTE PROCEDURE bucardo_add_delta_role() bucardo_triggerkick_a_delta AFTER INSERT OR DELETE OR UPDATE ON test FOR EACH STATEMENT EXECUTE PROCEDURE bucardo_triggerkick_a_delta()

endpoint.com

TRIGGER

Flow of control

NOTIFY MCP CTLNOTIFY

INSERT INTO “q”

KID

SELECT from “q”UPDATE “q”

REPLICATIONEVENT!

NOTIFY

Bucardo

endpoint.com

How Syncs happen

• NOTIFY vs. timeout vs. kick

• NOTIFY - from master db to MCP

• timeout - controller detects

• kick - manual/from MCP to controller

endpoint.com

Processes!• MCP, CTL, KID

• MCP: master control process

• CTL: controller (track, kick off/kill KIDs)

• KID: sync processes

endpoint.com

Administrativa

• Totally ok to kill KIDs

• CTL cleans up after you

• May need to kill open database handles (for unfinished COPYs, etc)

endpoint.com

Other interesting tables

On your master DB:

• bucardo_delta table has: ID, txn_time

• bucardo_track table: who’s done what?

endpoint.com

Sync types

• Fullcopy. Timeout/kick. TRUNCATE or DELETE, then COPY

• Pushdelta. Trigger. One way master-slave.

• Swap. Trigger. Two-way. Conflict resolution.

endpoint.com

Note about triggers:

• canonical way to disable triggers: ALTER TABLE DISABLE TRIGGER ALL;:(

• pre 8.3 way: UPDATE pg_class..

• SET session_replication_role = ‘replica’;

• Applies to just the current session

• Thanks, Jan!

endpoint.com

Dealing with failure

• Troubleshooting

• Backing up Bucardo itself

• What happens when you lose the Bucardodb?

endpoint.com

Bucardo hooks

• LIVE DATA CHANGES!

• pass in a hashref (ROW or ROWINFO)

• Returned value - bitmapped (a/b or both!)

• Affect the replication event, side effects

• Example: cache invalidation

endpoint.com

Bucardo Hooks

• customcode & customcode_map

• Conflict handlers (SWAP syncs)

• Pure Perl

• Exception - fix and try again

• Pre and post trigger

endpoint.com

Development

• bucardo.org

• mailing lists: http://bucardo.org/wiki/Bucardo_mailing_list

• Release early, often.

• Please submit patches! bug reports! tests!

endpoint.com

Questions?

endpoint.com

Thanks!

http://www.slideshare.net/selenamarie/bucardo

selena@endpoint.com

@selenamarie on Twitter

rss: http://www.chesnok.com/daily