postgresql replication and some test scenarios
TRANSCRIPT
![Page 1: Postgresql Replication and Some Test Scenarios](https://reader035.vdocument.in/reader035/viewer/2022062412/587b373e1a28ab057d8b747d/html5/thumbnails/1.jpg)
POSTGRESQL REPLICATIONNUR AGUS SURYOKO
![Page 2: Postgresql Replication and Some Test Scenarios](https://reader035.vdocument.in/reader035/viewer/2022062412/587b373e1a28ab057d8b747d/html5/thumbnails/2.jpg)
SIMPLICTIC WAY TO REPLICATE POSTGRES
Primary DB Standby DB
1. Backup 3. RESTORE2. RSYNC
![Page 3: Postgresql Replication and Some Test Scenarios](https://reader035.vdocument.in/reader035/viewer/2022062412/587b373e1a28ab057d8b747d/html5/thumbnails/3.jpg)
THIS METHOD IS… It work, yes• But:• With database growing, replication
getting slower• When the duration of backup +
rsync + restore > the interval = disaster
• Rely heavily on scripts
![Page 4: Postgresql Replication and Some Test Scenarios](https://reader035.vdocument.in/reader035/viewer/2022062412/587b373e1a28ab057d8b747d/html5/thumbnails/4.jpg)
PROPOSAL
PostgresReplicationAsynchronousHot-Stanby
![Page 5: Postgresql Replication and Some Test Scenarios](https://reader035.vdocument.in/reader035/viewer/2022062412/587b373e1a28ab057d8b747d/html5/thumbnails/5.jpg)
IN ONE SENTENCE
Same with Oracle Dataguard and DB2-HADR
![Page 6: Postgresql Replication and Some Test Scenarios](https://reader035.vdocument.in/reader035/viewer/2022062412/587b373e1a28ab057d8b747d/html5/thumbnails/6.jpg)
AT A GLANCE
Primary DB Standby DB
Archive-Log Archive-Log
1. SQL Connection Open
2. Restore Database + Copy Archive Log
3. Replication in Sync
![Page 7: Postgresql Replication and Some Test Scenarios](https://reader035.vdocument.in/reader035/viewer/2022062412/587b373e1a28ab057d8b747d/html5/thumbnails/7.jpg)
REMEMBER DB2-HADR? JUST THE SAME
The asynchronous one
![Page 8: Postgresql Replication and Some Test Scenarios](https://reader035.vdocument.in/reader035/viewer/2022062412/587b373e1a28ab057d8b747d/html5/thumbnails/8.jpg)
YEAH, BABY!
READ-ONLY QUERY ON STANDBY
Same capability with Oracle Active Dataguard and DB2-HADR RoS (Read on Standby)
![Page 9: Postgresql Replication and Some Test Scenarios](https://reader035.vdocument.in/reader035/viewer/2022062412/587b373e1a28ab057d8b747d/html5/thumbnails/9.jpg)
LET’S GET TO WORK2 nodes Installation: Download EnterpriseDB PostgreSQL binary distribution, save a lot of workConventions:Node1 hostname: userverNode2 hostname: userver2
Create sample table:CREATE TABLE test (
id uuid DEFAULT uuid_generate_v4() NOT NULL,
waktu timestamp with time zone DEFAULT now() NOT NULL,
text character varying(255)
);
![Page 10: Postgresql Replication and Some Test Scenarios](https://reader035.vdocument.in/reader035/viewer/2022062412/587b373e1a28ab057d8b747d/html5/thumbnails/10.jpg)
CONTINUOUSCreate this script:#!/bin/bashwhile :do psql -d test -c "insert into test(text) values('X')"; sleep 1;done;
Run this script on node-1:primaryYou see what we are trying to do here…
![Page 11: Postgresql Replication and Some Test Scenarios](https://reader035.vdocument.in/reader035/viewer/2022062412/587b373e1a28ab057d8b747d/html5/thumbnails/11.jpg)
NETWORKS• Maintain /etc/hosts on each nodes so that they can ping
each other• Configure ssh-key for user postgres on each nodes. Allow
passwordless ssh
![Page 12: Postgresql Replication and Some Test Scenarios](https://reader035.vdocument.in/reader035/viewer/2022062412/587b373e1a28ab057d8b747d/html5/thumbnails/12.jpg)
PRIMARY SETUPpsql# create user replicator replication login encrypted password 'Initial1';
Keep the name. We don’t want any trouble. Believe me, I tried
![Page 13: Postgresql Replication and Some Test Scenarios](https://reader035.vdocument.in/reader035/viewer/2022062412/587b373e1a28ab057d8b747d/html5/thumbnails/13.jpg)
PRIMARY SETUPedit postgresql.conf:wal_level = hot_standbyfsync = onsynchronous_commit = localwal_sync_method = fsyncwal_compression = onarchive_mode = onarchive_command = 'test ! -f /opt/postgresql/9.5/backup/archive/%f && cp %p /opt/postgresql/9.5/backup/archive/%f'max_wal_senders = 8wal_keep_segments = 24wal_sender_timeout = 10shot_standby = on
Careful with reddy things, adjust accordingly
![Page 14: Postgresql Replication and Some Test Scenarios](https://reader035.vdocument.in/reader035/viewer/2022062412/587b373e1a28ab057d8b747d/html5/thumbnails/14.jpg)
PRIMARY SETUPas root: mkdir -p /opt/postgresql/9.5/backup/archiveas root: mkdir -p /opt/postgresql/9.5/backupas root: chown -R postgres:postgres /opt/postgresql/9.5/backuprestart postgres primaryensure no error in $PGDATA/pg_log
![Page 15: Postgresql Replication and Some Test Scenarios](https://reader035.vdocument.in/reader035/viewer/2022062412/587b373e1a28ab057d8b747d/html5/thumbnails/15.jpg)
PRIMARY SETUPedit pg_hba.conf:host all replicator 192.168.56.105/32 md5host replication replicator 192.168.56.105/32 md5restart postgres primaryensure no error in $PGDATA/pg_log
Why 2? Because replication is not a database. It is a special keyword
![Page 16: Postgresql Replication and Some Test Scenarios](https://reader035.vdocument.in/reader035/viewer/2022062412/587b373e1a28ab057d8b747d/html5/thumbnails/16.jpg)
STANDBY SETUPtest connection to primary:psql -h userver -d test -U replication -W
![Page 17: Postgresql Replication and Some Test Scenarios](https://reader035.vdocument.in/reader035/viewer/2022062412/587b373e1a28ab057d8b747d/html5/thumbnails/17.jpg)
STANDBY SETUPedit postgresql.conf:wal_level = hot_standbyfsync = onsynchronous_commit = localwal_sync_method = fsyncwal_compression = onarchive_mode = onarchive_command = 'test ! -f /opt/postgresql/9.5/backup/archive/%f && cp %p /opt/postgresql/9.5/backup/archive/%f'max_wal_senders = 8wal_keep_segments = 24wal_sender_timeout = 10shot_standby = on
![Page 18: Postgresql Replication and Some Test Scenarios](https://reader035.vdocument.in/reader035/viewer/2022062412/587b373e1a28ab057d8b747d/html5/thumbnails/18.jpg)
STANDBY SETUPdestroy db on standbyroot: rm -rf /opt/postgresql/9.5/dataroot: mkdir -p /opt/postgresql/9.5/dataroot: chown -R postgres:postgres /opt/postgresql/9.5/dataroot: chmod -R 700 /opt/postgresql/9.5/data
Destroy db, not the engine
backup - and direct restore from primary:pg_basebackup -h userver -D /opt/postgresql/9.5/data -U replicator -v -P
![Page 19: Postgresql Replication and Some Test Scenarios](https://reader035.vdocument.in/reader035/viewer/2022062412/587b373e1a28ab057d8b747d/html5/thumbnails/19.jpg)
STANDBY SETUPprepare file /opt/postgresql/9.5/data/recovery.conf:standby_mode = 'on'primary_conninfo = 'host=userver port=5432 user=replicator password=Initial1'trigger_file = '/home/postgres/trigger.file‘
start standbycheck test table. data should be updated periodically
![Page 20: Postgresql Replication and Some Test Scenarios](https://reader035.vdocument.in/reader035/viewer/2022062412/587b373e1a28ab057d8b747d/html5/thumbnails/20.jpg)
STANDBY SETUPcheck replication status:test=# select * from pg_stat_replication;sync_state must be ‘sync’
![Page 21: Postgresql Replication and Some Test Scenarios](https://reader035.vdocument.in/reader035/viewer/2022062412/587b373e1a28ab057d8b747d/html5/thumbnails/21.jpg)
SYNC
![Page 22: Postgresql Replication and Some Test Scenarios](https://reader035.vdocument.in/reader035/viewer/2022062412/587b373e1a28ab057d8b747d/html5/thumbnails/22.jpg)
TEST1: NODE2 SHUTDOWN OSnode1: primarynode2: standbyshutdown node2transaction on primary still goingreplication status:test=# select * from pg_stat_replication;<empty>
Test Result: OK
![Page 23: Postgresql Replication and Some Test Scenarios](https://reader035.vdocument.in/reader035/viewer/2022062412/587b373e1a28ab057d8b747d/html5/thumbnails/23.jpg)
TEST2: START STANDBYnode1: primarynode2: standbystartup node2auto-start after OS, and sync still goingcheck replication status:test=# select * from pg_stat_replication;sync_state = ‘sync’
Test Result: OK
![Page 24: Postgresql Replication and Some Test Scenarios](https://reader035.vdocument.in/reader035/viewer/2022062412/587b373e1a28ab057d8b747d/html5/thumbnails/24.jpg)
TEST3: NORMAL FAILOVERnode1: primarynode2: standbystop periodic insertmark last inserted value (primary): 2668fadd-59dc-468a-ae85-65f9750bc336 | 2015-11-05 22:04:15.658038+07 | Xshutdown node1
![Page 25: Postgresql Replication and Some Test Scenarios](https://reader035.vdocument.in/reader035/viewer/2022062412/587b373e1a28ab057d8b747d/html5/thumbnails/25.jpg)
TEST3: NORMAL FAILOVERfound disconnected log in standby, this is normal:2015-11-05 22:14:18 WIB FATAL: could not connect to the primary server: could not connect to server: Connection refusedIs the server running on host "userver" (192.168.56.104) and acceptingTCP/IP connections on port 5432?
![Page 26: Postgresql Replication and Some Test Scenarios](https://reader035.vdocument.in/reader035/viewer/2022062412/587b373e1a28ab057d8b747d/html5/thumbnails/26.jpg)
TEST3: NORMAL FAILOVERnode2 as primarytouch /home/postgres/trigger.filecheck log, this is a successful failover message:2015-11-05 22:15:53 WIB LOG: trigger file found: /home/postgres/trigger.file2015-11-05 22:15:53 WIB LOG: redo done at 0/100000282015-11-05 22:15:53 WIB LOG: last completed transaction was at log time 2015-11-05 22:04:15.658499+072015-11-05 22:15:53 WIB LOG: selected new timeline ID: 22015-11-05 22:15:54 WIB LOG: archive recovery complete2015-11-05 22:15:54 WIB LOG: MultiXact member wraparound protections are now enabled2015-11-05 22:15:54 WIB LOG: database system is ready to accept connections2015-11-05 22:15:54 WIB LOG: autovacuum launcher started
![Page 27: Postgresql Replication and Some Test Scenarios](https://reader035.vdocument.in/reader035/viewer/2022062412/587b373e1a28ab057d8b747d/html5/thumbnails/27.jpg)
TEST3: NORMAL FAILOVERcheck latest inserted data, and compare to marked data:
2668fadd-59dc-468a-ae85-65f9750bc336 | 2015-11-05 22:04:15.658038+07 | X
data match!
ensure /home/postgres/trigger.file is deleted automatically by postgres
Test Result: OK
![Page 28: Postgresql Replication and Some Test Scenarios](https://reader035.vdocument.in/reader035/viewer/2022062412/587b373e1a28ab057d8b747d/html5/thumbnails/28.jpg)
TEST4: NETWORK DISCONNECTnode1: primary
node2: standby
ensure the database is in sync
cut network connection
node1: primary: not in sync
node2: cannot connect to primary
re-connect network
node1: sync
node2: log report started streaming WAL
Test Result: OK
![Page 29: Postgresql Replication and Some Test Scenarios](https://reader035.vdocument.in/reader035/viewer/2022062412/587b373e1a28ab057d8b747d/html5/thumbnails/29.jpg)
TEST5: NODE1 SHUTDOWN node1: primary
node2: standby
shutdown node1: primary
startup node1: primary
after startup, automatically sync
check test database, updated periodically
Test Result: OK
![Page 30: Postgresql Replication and Some Test Scenarios](https://reader035.vdocument.in/reader035/viewer/2022062412/587b373e1a28ab057d8b747d/html5/thumbnails/30.jpg)
THANK YOU